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Preface 



For the thirtieth year, the Research and Theory Division of the Association for Educational 
Communications and Technology (AECT) is sponsoring the publication of these Proceedings. Papers 
published in this volume were presented at the national AECT Convention in Anaheim, CA. A limited 
quantity of these Proceedings were printed and sold in both hardcopy and electronic versions. Volume #1 is 
available through the Educational Resources Clearinghouse (ERIC) System. Proceedings volumes are 
available to members at AECT.ORG. 

The Proceedings of AECT’s Convention are published in two volumes. Volume #1 contains papers dealing 
primarily with research and development topics. Papers dealing with instruction and training issues are 
contained in volume #2. 

REFEREEING PROCESS: Papers selected for presentation at the AECT Convention and included in these 
Proceedings were subjected to a reviewing process. All references to authorship were removed from 
proposals before they were submitted to referees for review. Approximately sixty percent of the 
manuscripts submitted for consideration were selected for presentation at the convention and for 
publication in these Proceedings. The papers contained in this document represent some of the most current 
thinking in educational communications and technology. 
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Motivation and Self-Regulation in Online Courses: 

A Comparative Analysis of Undergraduate and Graduate Students 



Anthony R. Artino, Jr. and Jason M. Stephens 
University of Connecticut 

Abstract 

This study provides a comparative analysis of undergraduate and graduate students’ academic motivation and self- 
regulation while learning online. Participants (N= 82) completed a survey that assessed several experiential, 
motivational, and self-regulatory factors. As hypothesized, graduate students reported higher levels of critical 
thinking than undergraduates. Moreover, after controlling for experiential differences, logistic regression analyses 
indicated that graduate student membership was predicted by higher levels of critical thinking, lower levels of 
procrastination, and lower intentions to enroll in future online courses. Implications for online instructors and 
suggestions for future research are discussed. 



Background 

Online learning 1 has become the format-of-choice for numerous postsecondary institutions eager to provide 
students with the opportunity to learn from a distance (Bernard et al., 2004; Larreamendy-Joems & Leinhardt, 2006; 
Moore & Kearsley, 2005; Tallent-Runnels et al., 2006). Evidence of the explosive growth in online learning is not 
difficult to find. For instance, a recent survey of 2,200 U.S. colleges and universities by the Sloan Consortium 
(2006) found that 96% of large institutions (greater than 15,000 total enrollments) have some online offerings; 62% 
of Chief Academic Officers rated learning outcomes in online education as the same or superior to traditional, face- 
to-face instruction; 58% of schools identified online education as a critical long-term strategy; and overall online 
enrollment increased from 2.4 million in 2004 to 3.2 million in 2005. 

As online learning has grown, so too has interest in students’ academic motivation and self-regulation 
(Greene & Azevedo, 2007). Self-regulated learners are generally characterized as active participants who efficiently 
control their own learning experiences in many different ways, including organizing and rehearsing information to 
be learned; monitoring their thinking processes and seeking help when they do not understand; and holding positive 
motivational beliefs about their capabilities and the value of learning (Boekaerts, Pintrich, & Zeidner, 2000; Schunk 
& Zimmerman, 1998). Self-regulated learning (SRL) — sometimes referred to as academic self-regulation — has also 
been described as an active, constructive process whereby students set goals for their learning based on past 
experiences and the contextual features of the current environment (Pintrich, 2000). These learning goals then 
become the standards against which academic progress is compared (Greene & Azevedo, 2007). It is important to 
note, however, that academic self-regulation is not an all-or-nothing phenomenon. Instead, students are self- 
regulating to the extent that they are cognitively, motivationally, and behaviorally involved in their own learning 
activities (Zimmerman, 2000). 

Recently, several scholars (e.g., Azevedo, 2005; Dabbagh & Bannan-Ritland, 2005; Dabbagh & Kitsantas, 
2004) have suggested that to be successful in highly autonomous online learning situations, students may require 
well-developed SRL skills to guide their cognition and behavior. Moreover, some researchers (Greene & Azevedo, 
2007; Pintrich, 2003; Schunk, Pintrich, & Meece, 2008) have indicated that there may be important developmental 
differences in students’ self-regulatory skills, differences that warrant further empirical investigation. For example, 
Greene and Azevedo (2007) have encouraged researchers to ask whether there might be a developmental 
progression within SRL. In their words, “research in this area would perhaps not only allow us to more clearly 
examine individual phenomena in SRL but also provide clues as to how good SRL behaviors might be taught” 
(Greene & Azevedo, 2007, p. 364). What’s more, such developmental differences, if they do exist, could have 
important educational implications for instructors, determining, for example, the cognitive demands of learning 
activities faculty design, as well as type and level of scaffolding they provide during instruction. 



1 Online learning is commonly referred to as online education, Web-based learning, or Web-based education (Zhao, Lei, Yan, 
Lai, & Tan, 2005). 
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Purpose of the Study 



The purpose of the present study was to begin exploring potential developmental differences in academic 
self-regulation, as described by several scholars in the field of academic motivation (e.g., Greene & Azevedo, 2007; 
Pintrich, 2003; Schunk et ah, 2008). In particular, this study was designed to determine if there are experiential, 
motivational, and self-regulatory differences between undergraduate and graduate students enrolled in several online 
courses. We hypothesized that graduate students would exhibit more adaptive SRL profiles than their undergraduate 
counterparts, due, in part, to their greater experience as learners at the university level. Ultimately, identifying such 
differences could help faculty as they attempt to employ effective online teaching strategies for students who may 
have varying levels of academic motivation and diverse self-regulatory capacities. 

Method 



Participants 

Participants for this study included a convenience sample of 82 students from a large public university in 
the northeastern United States. Of these students, 39 (48%) were undergraduates and 43 (52%) were graduate 
students. Participants were enrolled in several different courses delivered completely online through WebCT. The 
sample included 39 women (48%) and 43 men (52%). The mean age of the undergraduate participants was 22.9 
years (SD = 2.5; range 19-29), and the mean age of the graduate students was 31.2 years (* SD = 9.0; range 21-56). 

Overall, participants reported a wide range of educational experience. In particular, undergraduates 
reported the following: High School ( n = 3, 8%), Some College ( n = 23, 59%), 2-Year College Degree (; n = 8, 20%), 
and 4-Year College Degree ( n = 5, 13%). In contrast, graduate students reported the following: 4-Year College 
Degree (n = 16, 37%), Master’s Degree ( n = 26, 61%), and Professional Degree (n = 1, 2%). In terms of experience 
with online learning, 36 undergraduates (92% of undergraduates) reported that they had completed one or more 
online courses in the past, whereas only 23 graduate students (53% of graduates) reported the same level of 
experience with online learning. 

Procedures and Instrumentation 

During the last four weeks of the semester, participants completed an anonymous, online survey. The first 
part of the survey was composed of 32 items; all items employed a 7-point Likert-type response scale ranging from 
1 (< completely disagree) to 7 (< completely agree). The items in this section were further subdivided into six subscales, 
and all of the variables derived from this part of the survey were created by computing means of the items associated 
with a particular subscale. The six subscales included the following: 

Motivational Beliefs (adapted from the Motivated Strategies for Learning Questionnaire [MSLQ]; Pintrich, 

Smith, Garcia, & McKeachie, 1993): 

1 . Task Value - students’ judgments of how interesting, useful, and important the course content was 
to them (6 items, a = .92); 

2. Self-Efficacy for Learning and Performance - students’ perceptions of expectancy for success and 
confidence in their ability to perform the learning task (7 items, a = .91); 2 

SRL Strategies (adapted from the MSLQ; Pintrich et al, 1993): 

3. Elaboration - students’ use of elaboration strategies (e.g., paraphrasing, summarizing; 5 items, a = 
.87); 

4. Critical Thinking - students’ use of critical thinking strategies (e.g., applying previous knowledge 
to new situations or making critical evaluations of ideas; 5 items, a = .87); 

Motivational Engagement (adapted from Wolters, 2003, 2004): 

5. Procrastination - students’ level of academic disengagement or tendency to put off getting started 
on the work required for their online course (5 items, a = .90); and, 

6. Choice - students’ intentions to enroll in future online courses (4 items, a = .88). 

The second part of the survey was composed of background and demographic items, including two 
individual items that assessed students’ online technologies experience and previously completed online courses. 
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The definition of self-efficacy used to develop the MSLQ’s self-efficacy scale is a bit broader than other measures, which 
usually limit themselves to assessing confidence in one’s ability to attain designated types of performances and do not include 
expectancy for success (see discussion in Duncan & McKeachie, 2005). 
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Results 



Descriptive Statistics 

Table 1 presents descriptive statistics for the eight variables measured in the present study. As indicated, 
five of the six subscale variables (i.e., task value, self-efficacy, elaboration, critical thinking, and choice) had means 
slightly above the midpoint of the response scale (5.63, 5.87, 5.62, 5.14, and 4.57, respectively) and standard 
deviations ranging from 0.92 to 1.73. Additionally, all five variables showed a slight negative skew. On the other 
hand, descriptive statistics for the procrastination variable indicated a mean just below the midpoint of the response 
scale (3.32) and a standard deviation of 1.61. The frequency distribution for the procrastination variable had a slight 
positive skew. 

Group Comparisons: T Tests 

Independent samples t tests were conducted to explore differences between undergraduate and graduate 
students on the eight variables measured. To control the type I error rate, a Bonferroni adjustment was used (alpha = 
.05/8 = .008). Results from these analyses, also presented in Table 1, revealed statistically significant group 
differences on four of the eight variables. As hypothesized, graduate students reported higher levels of critical 
thinking than undergraduates (p < .008; d = -0.68). Undergraduates, by contrast, reported having completed more 
online courses in the past (p < .001; d= 1.38); more experience with online technologies (p < .008; d = 0.61); and 
greater intentions to enroll in additional online courses in the future (p < .008; d = 0.84). Effect sizes for the four 
statistically significant findings were moderate to large (Cohen, 1988). 



Table 1 

Means (Standard Deviations) and Independent Sample t and Cohen’s d Statistics for the Eight Measured Variables 



Variable 


Overall 
(N = 82) 


Group 

Undergraduate 
(n = 39) 


Graduate 
(n = 43) 


^-Statistic 


Cohen’s d 


Experience 


Online Tech. Experience 


5.98 (1.25) 


6.36(1.31) 


5.63 (1.09) 


2.76* 


0.61 


No. Completed Online Courses 


3.37 (3.29) 


5.13(3.96) 


1.77 (1.07) 


5.13* 


1.38 


Motivational Beliefs 


Task Value 


5.63(1.16) 


5.81 (1.16) 


5.47(1.14) 


1.35 


0.30 


Self-Efficacy 


5.87 (0.92) 


5.65 (0.99) 


6.06 (0.82) 


-2.02 


-0.46 


SRL Strategies 


Elaboration 


5.62 (1.23) 


5.46 (1.36) 


5.76 (1.09) 


-1.11 


-0.25 


Critical Thinking 


5.14(1.28) 


4.71 (1.33) 


5.53(1.10) 


-3.03* 


-0.68 


Motivational Engagement 


Procrastination 


3.32 (1.61) 


3.75 (1.64) 


2.92 (1.50) 


2.42 


0.53 


Choice 


4.57 (1.73) 


5.28 (1.56) 


3.93 (1.64) 


3.80* 


0.84 



Note. Bonferroni adjustment was used to control for inflation of type I error associated with multiple comparisons: alpha = .05/8 = 
.008. Cohen’s d = (M 2 - M0 / V [(oi 2 + o 2 2 )/2]. The online technologies variable was measured on a 7-point Likert-type response 
scale ranging from 1 (extremely inexperienced) to 7 ( extremely experienced). The number of completed online courses ranged from 
1 to 17. The remaining variables were measured on a 7-point, Likert-type agreement scale. 

*p < .008. 
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Group Comparisons: Logistic Regression 

Logistic regression was used to investigate the unique contribution of these differences in predicting 
students’ group membership (undergraduate = 0, graduate student =1). Using a hierarchical method, the 
independent variables were grouped into four construct sets and entered into the equation as follows: Step 1, 
experiential variables (online technologies and online learning experience); Step 2, motivational beliefs (task value 
and self-efficacy); Step 3, SRL strategies (elaboration and critical thinking); and Step 4, motivational engagement 
(procrastination and choice). 

Table 2 provides a summary of the hierarchical logistic regression. As indicated, model fit statistics 
improved with the addition of each construct set. In the final model, 93.9% of students were correctly classified (-2 
log likelihood = 21.00; x 2 (8) = 92.48, p < .001), and the likelihood ratio R-square (i.e., the proportional reduction in 
deviance produced by final model when compared to the null model; Menard, 2000) was large (.81). Additionally, 
four variables were statistically significant predictors of group membership: online technologies experience (Z? = - 
3.48 , p < .05), online learning experience ( b = -1.99, p< .05), procrastination (b = -1.59,/? < .05), and choice (b = - 
1.24,/? < .05). The critical thinking variable approached statistical significance (b = 2.81,/? = .058). 



Table 2 

Model Summaries for the Hierarchical Logistic Regression Predicting Group Membership (undergraduate = 0; graduate = 1) 







Step 1 






Step 2 






Step 3 






Step 4 




Variables 


b 


SE 


OR 


b 


SE 


OR 


b 


SE 


OR 


b 


SE 


OR 


Experience 


























Online Tech. 
Experience 


-1.34** 


0.41 


0.26 


-1.85** 


0.58 


0.16 


-2.26** 


0.69 


0.15 


-3.48* 


1.66 


0.03 


No. Completed 
Online Courses 


-1.27*** 


0.31 


0.28 


-1 .43*** 


0.37 


0.24 


-1.61*** 


0.45 


0.20 


-1.99* 


0.80 


0.14 


Motivational 

Beliefs 


























Task Value 








-0.69 


0.44 


0.50 


-1.24 


0.67 


0.29 


-2.71 


1.48 


0.07 


Self-Efficacy 








1 .67** 


0.55 


5.31 


1.04 


0.74 


2.83 


2.82 


1.73 


16.70 


SRL Strategies 


























Elaboration 














-0.57 


0.76 


0.95 


-1.01 


1.76 


0.37 


Critical Thinking 














1.63* 


0.70 


5.11 


2.81 a 


1.48 


16.67 


Motivational 

Engagement 


























Procrastination 




















-1.59* 


0.69 


0.21 


Choice 




















-1.24* 


0.53 


0.29 


Fit Statistics 


























-2 Log 
Likelihood 




60.67 






47.82 






37.04 






21.00 




Model Likelihood 
Ratio x 2 




52.81*** 






65.66*** 






76.44*** 






92.48*** 




Block Likelihood 
Ratio x 




52.81*** 






12.85** 






10.78** 






16.04*** 




Model Likelihood 
Ratio R 2 




.48 






.58 






.67 






.81 




% Correctly 
Classified 




86.6 






86.6 






89.0 






93.9 





Note, b = unstandardized regression coefficient; OR = odds ratio. 

a Wald x 2 test for the critical thinking variable approached statistical significance (p = .058). 
*p < .05. **p < .01 . ***p < .001 . 
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Next, a logistic regression was conducted using only the four statistically significant variables, as well as 
the critical thinking variable, as predictors of group membership. Table 3 presents results from this more 
parsimonious, five-predictor model. As indicated, 91.5% of students were correctly classified (-2 log likelihood = 
29.10; x 2 (5) = 84.38,/) < .001), and the likelihood ratio R-square was large (.74). Moreover, all five variables were 
statistically significant predictors of group membership: online technologies experience ( b = -2.73,/? < .01), online 
learning experience (b = -1.1 1, p < .01), critical thinking (b = 1.74,/? < .01), procrastination ( b = -0.89,/? < .05), and 
choice (b = -0.82,/? < .05). 

Results from the final model with five predictors indicated that the odds of graduate student membership 
were higher as one’s use of critical thinking strategies increased, and lower as one’s online technologies experience, 
number of completed online courses, tendency to procrastinate in one’s current online course, and intentions to 
enroll in future online courses increased. In others words, after controlling for experiential differences, graduate 
students were more likely to use critical thinking strategies during online learning; whereas undergraduates were 
more likely to procrastinate in their online courses and, paradoxically, were more likely to report wanting to take 
more online courses in the future. 



Table 3 

Model Summary for the Logistic Regression Model with Five Independent Variables Predicting Group Membership (undergraduate = 
0; graduate = 1) 













Model Fit Statistics 




Variable 


B 


SE 


OR 


-2 Log 
Likelihood 


Likelihood 
Ratio x 


Likelihood 
Ratio R 2 


% Correctly 
Classified 


Online Tech. 
Experience 


-2.73** 


0.86 


0.07 










No. Completed 
Online Courses 


-1.71** 


0.54 


0.18 










Critical Thinking 


1.74** 


0.66 


5.70 


29.10 


84.38*** 


.74 


91.5 


Procrastination 


-0.89* 


0.36 


0.41 










Choice 


-0.82* 


0.34 


0.44 











Note, b = unstandardized regression coefficient; OR = odds ratio. 
*p < .05. **p < .01 . ***p < .001 . 



Discussion 

Findings from this comparative study reveal that undergraduate and graduate students learning online differ 
in a number of important ways. Taken together, results partially support the hypothesis that graduate students exhibit 
more adaptive SRL profiles. Specifically, though less experienced with online technologies and learning, graduate 
students reported greater use of critical thinking strategies and lower levels of procrastination. These latter 
characteristics are consistent with effective academic self-regulation (Pintrich, 1999; Wolters, 2003). Interestingly, 
undergraduates, who reported greater levels of procrastination in their current online courses, also expressed greater 
intentions to enroll in future online courses. This paradoxical finding was not anticipated and may warrant further 
investigation. 

Implications for Online Instructors 

Results from the present study suggest some preliminary implications for online instructional practice. In 
particular, findings indicate that, as Greene and Azevedo (2007) have suggested, there may be a developmental 
progression within academic self-regulation. With this result in mind, online instructors might consider providing 
their undergraduate and graduate students with differential support; that is, different types and amounts of regulatory 
guidance and scaffolding during online learning activities. Specifically, the following suggestions for online 
instructional practice are provided: 

7. Provide explicit instructional support and structure. Although, as this study revealed, undergraduates 
may be more experienced with online technologies and learning, they may also require more explicit support and 
structure from the instructor, as indicated by their lower levels of critical thinking and greater tendency to 
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procrastinate. Examples of explicit support and structure include reflective prompting aimed at helping students self- 
monitor their understanding (Davis & Linn, 2000); clear and detailed syllabi and assignment instructions; and more 
intermediate assignment deadlines to facilitate task completion (Liu, Bonk, Magjuka, Lee, & Su, 2005). In general, 
these instructional tactics are designed to encourage learners to better regulate their own learning in online contexts 
(McLoughlin, 2002) and to discourage their use of maladaptive academic behaviors, such as procrastination. 

2. Develop students ’ self-efficacy. Another approach to encouraging self-regulation and discouraging 
procrastination is to address students’ self-efficacy for learning online. Research with college undergraduates in 
traditional classrooms has indicated that students with higher self-efficacy tend to procrastinate less often than others 
(Wolters, 2003). Thus, although undergraduates in the present study reported similar levels of self-efficacy as 
graduate students, interventions aimed at promoting self-efficacy in online settings may be an effective means of 
reducing procrastination. Two instructional strategies that have been known to enhance students’ self-efficacy for 
learning in both traditional and online contexts include helping learners identify and set challenging, proximal goals 
(Locke & Latham, 1990; Dabbagh & Kitsantas, 2004) and providing students with timely, honest, and explicit 
performance feedback (Bandura, 1997; Bangert, 2004; Wang & Lin, 2007). 

3. Encourage collaboration and co-regulation. Many models of self-regulation support the idea of external 
regulation from teachers and peers as they provide modeling of and scaffolding for regulatory behaviors (Boekaerts 
et al., 2000). In a collaborative online environment, this type of external regulation might be particularly effective. 
For example, by requiring students to work together toward a mutual goal, online instructors can encourage students 
to provide regulatory support for one another in the form of project planning, monitoring, and reflecting (Winters & 
Azevedo, 2005). This type of collaborative regulation has been called co-regulation (Como & Randi, 1999), and 
research in traditional and hypermedia learning environments has revealed that, “under these circumstances, 
students’ individual self-regulatory processes are mediated by the collaborative context in which they are learning” 
(Winters & Azevedo, 2005, p. 193). 

Of course, simply placing students in groups does not guarantee collaboration and co-regulation (Johnson 
& Johnson, 1999). Instead, the learning environment must be intentionally designed to promote effective group 
behaviors and to discourage maladaptive activities such as free-riding and social loafing (Kreijns, Kirschner, & 
Jochems, 2003). Although a discussion of specific techniques for promoting effective collaboration is beyond the 
scope of this article, these instmctional methods have been detailed elsewhere in the computer-supported, 
collaborative learning literature (see, for example, Hiltz, 1997; Johnson & Johnson, 1999). 

4. Scaffold online discussions. A primary goal of online discussions is to encourage students to challenge, 
reform, and synthesize their current views of knowledge through in-depth interactions with others (Garrison, 
Anderson, & Archer, 2001). However, findings from numerous studies of online discussion fomms have indicated 
that students’ interactions are often quite shallow, and “rarely developed into a higher level of communication where 
negotiation, co-constmction, and agreement occurred” (Tallent-Runnels et al., 2006, p. 100). One possible 
explanation for students’ shallow participation in online discussions is lack of guidance from the instmctor. Thus, as 
Christopher, Thomas, and Tallent-Runnels (2004) have argued, online instmctors, like their counterparts in 
traditional classrooms, must take greater responsibility for organizing and scaffolding their students’ learning within 
these online discussions. 

In the present study, undergraduates reported lower levels of critical thinking than graduate students. This 
finding suggests that online instmctors may need to provide additional scaffolding for these students in an effort to 
enhance their use of critical thinking skills and other deep processing strategies. This type of supplemental 
scaffolding in online contexts has been described by some as enhanced teaching presence (Anderson, Rourke, 
Garrison, & Archer, 2001; Garrison et al., 2001; Shea, Swan, Li, & Picket, 2005). For example, during online 
discussions, enhanced teaching presence might include some of the following teacher behaviors: setting the climate 
for learning by modeling appropriate discussion posts; focusing the discussion on specific issues; encouraging, 
acknowledging, and reinforcing student contributions; identifying areas of agreement/disagreement and seeking 
consensus and understanding; adding information from diverse sources to a string of student posts; critically 
evaluating posts and requesting clarification and elaboration where necessary; and diagnosing and correcting 
students’ misunderstandings (Anderson et al., 2001; Shea et al., 2005). Ultimately, teaching practices such as these 
that facilitate productive discourse may be necessary if online learners — particularly undergraduates — are expected 
to engage their classmates in meaningful interactions, develop higher levels of critical thinking, and realize other 
educationally worthwhile learning outcomes (e.g., reduced procrastination, improved persistence, and overall 
satisfaction with online learning; Christopher et al., 2004; Liu et al., 2005; Shea et al., 2005; Whipp, 2003). 
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Limitations and Future Directions 



One major limitation of the present study was the relatively small convenience sample utilized. Although 
significant differences were found in respondents’ experience and components of their academic self-regulation, the 
nature of the sample limits the extent to which these findings can be generalized to other university students. For 
instance, it is possible that these results are unique to the individuals surveyed and the specific online contexts 
investigated here (Shadish, Cook, & Campbell, 2002). Future research should include larger, more diverse samples 
to improve the external validity of these findings. 

Another limitation was the non-experimental nature of the present investigation. Specifically, it is unclear 
whether the higher levels of critical thinking reported by graduate students actually represents a developmental 
difference in these students when compared to undergraduates. Instead, these differences could have been the result 
of different course requirements; that is, by their very nature, graduate courses may simply require students to utilize 
more critical thinking strategies to successfully complete online learning activities. On the other hand, 
undergraduate courses may not require this type of cognitive processing, thus allowing undergraduates to get away 
with using more shallow processing strategies, such as rehearsal and rote memorization. If this is the case, then 
differences observed in the present study may be less about developmental differences in self-regulation and more 
about dissimilarity in the requirements inherent to graduate and undergraduate courses. That said, Christopher et al. 
(2004) found that higher level discussion prompts (i.e., prompts that modeled a high level of critical thinking) had 
no effect on the level of graduate students’ responses, as coded using a rubric developed from Bloom’s Taxonomy 
of Learning. Clearly, more controlled research is needed to further clarify the differences detected here. 

Finally, most of the instructional recommendations provided in this article require more empirical testing in 
online environments to validate their efficacy. Future studies that utilize experimental designs might be especially 
useful in exploring whether online interventions, such as adaptive scaffolding during online discussions, can 
differentially impact undergraduate and graduate students’ use of critical thinking and other deep processing 
strategies. The use of alternative research methods, such as content analysis of online discussion boards, could be 
particularly useful in exploring these relations. 



Conclusion 

Notwithstanding methodological limitations, results from the present study suggest that undergraduate and 
graduate students come to online courses with different levels of online experience and exhibit different levels of 
self-regulation while learning online. Moreover, these findings suggest that faculty should closely consider their 
online audience, as students’ experience, motivational beliefs, and self-regulatory competence should determine, in 
large part, the type and amount of structure, support, and scaffolding teachers provide during online instruction. 
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Abstract 

This presentation and associated paper focus on an education initiative that intends to help young 
learners (aged 8-16) in communities where there is limited access to technology in homes and schools acquire 
21 st century skills. The presentation has three parts: the first part focuses on introduction of the Intel® Learn 
program’s organization and content; the second part reveals the results of the program evaluations; and the third 
part covers the lessons learned during the implementation and evaluation of such a technology integration project 
in an emerging market. The results are quite promising. Although bureaucratic procedures, teachers’ old habits, 
preconceptions and students lack of previous experiences have caused some problems concerning 
implementation of the project, the Intel® Learn helped students acquire and/or improve technology, 
collaboration and critical thinking skills. Both quantitative and qualitative data showed that students’ acquisition 
of the technology skills exceeded the expected level and that they gained collaboration skills around the expected 
level. However, the staff should work more to improve students’ critical thinking skills. 

Introduction 

Over the last decade, a growing number of experts, not only from field of education but also from 
economics, politics, international relations and so forth, around the world has reached a consensus on a new set 
of skills that are needed to be able to survive in the 21st century and on providing learners alternative learning 
opportunities to learn more than just reading, writing, and arithmetic. The World Bank Group (2003) as well as 
Partnership for 21st Century Skills (2003) suggests that learners need to acquire critical thinking, effective 
communication, team work, continuous learning, and use of technology skills in order to help the global 
knowledge economy and be productive world citizens. Additionally experts, such as Kozma (2005a), express 
importance of a deeper understanding of core school subject including especially science, math, and technology 
and innovativeness in every aspect of life. On the other hand, helping learners acquire these skills requires a 
different instructional strategy then the traditional school approach that reflects not present day conditions but 
the past. Problem-based, technology-enhanced, authentic learning opportunities are considered as today’s 
instructional strategies (Jonassen et al, 2003). 

Kozma ( 2005b) provided a list of significant outcomes of use of ICT in education, including 
improving school attendance, deepening conceptual understanding in core school subjects, promoting wider 
involvement in community developments. Similarly SRI International (2006) notes that “enabling children to 
learn to use ICTs and . . . creating optimal opportunities for children to develop a fuller set of 21st-century skills” 
are the two main ways those promote 21st-century learning (p. 3). According to SRI International ICT 
integration programs should be designed carefully and provide authentic pathways to help novice learners find 
opportunities to gain 21st-century skills. 

Various projects that intended to help learners acquire these skills have been suggested, such as the 
Partnership for 21 st Century Skills, Skills: Getting on in Business, Getting on at Work, the Intel Education 
Initiative, etc. The Intel® Learn Program is one of these projects. 

Intel® Learn Program 

The Intel® Learn is an informal after-school program, designed for helping young learners (aged 8-16) 
in communities where there is limited access to technology in homes and schools acquire skills necessary to 
compete in the 21st Century: (1) technology literacy, (2) problem-solving skills and other forms of critical 
thinking, and (3) peer-to-peer collaboration. The program consists of a series of hands-on activities and projects 
that require children collaborate with fellow learners to accomplish authentic tasks, such as designing and 
developing a strategic action plan for the future of the learners’ own community, or for the possible future 
disasters in their region, and so forth. The Intel® Learn Program is part of the Intel Innovation in Education 
initiative, requires collaboration with local public and private educational agencies. After completing an 
evaluation of the program pilot in China, Israel, Mexico, and India, the Intel® Learn program has been expanded 
to some other countries including Egypt, Turkey, Brazil, and Russia. 

In Turkey, the Intel® Learn Program has been implemented in public schools since 2005. The Ministry 
of National Education (MEB) provides staff and logistics while Intel Foundation along with its partner 
institutions, such as the Institute of Computer Technology (ICT) and BilgeAdam, administer the program. After 
receiving training from BilgeAdam and MEB trainers, participant teachers offer the course as an after school 
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program. Up to January 2007, total 200 public school teachers were trained to put into practice the program in 
their schools. Around 180 teachers out of 200 offered the Intel® Learn in their schools at least one time (a batch) 
and trained more than 4000 students in 30 different provinces of the country. 

Effectiveness of the staff (teacher) trainings and the implementations in schools has been evaluated by a 
local of research organization (a third party along with MEB and Intel). The researchers in this organization have 
collaborated with the Center for Technology in Learning at SRI International that is a private for-profit 
organization specialized in evaluation and research, to conduct the evaluation. 

This paper gives details about the evaluation of the Intel® Learn Turkey Program. The reporting of data 
and results is organized into three sections. The first section elaborates the data collection. The second 
summarizes the findings on the following aspects of the program: (1) participation, (2) staff training, (3) 
usefulness of course materials, (4) learning outcomes, (5) staff-learner interactions, (6) program structure and 
organization, and (7) scale-up and future. The third section consists of our comments and recommendations 
about the effectiveness, efficiency, appeal and sustainability of the program. 

Data Collection 

A mixed method approach has been employed for the evaluation. Four survey instruments and two 
control lists helped researchers to collect quantitative data while semi- structured interviews with the teachers, 
school administrators, students and families, and class observations provided qualitative data. At the end of each 
staff training participant teachers were asked to fill out the End of Staff Training Survey. They were also asked 
to complete the Attendance Form, the Sample Works Form and the End of Education Survey. Additionally, they 
were required to send at least 15 works created by learners during each implementation. The evaluation team 
used a rubric (control list) to assess the quality of learners’ works they received form the teachers and an 
observation log (another control list) to see how the program was working for the learners in accomplishing the 
project’s goals (promoting technology skills, higher-order thinking, collaborative capabilities). 

The evaluation team has collected quantitative data from total 114 teachers (out of 200) during July 
2005- January 2007. The team also conducted 15 site (school) visits during this period. These visits help the team 
observe use of curriculum materials, staff-learner interactions, learner-learner interactions, and learning 
outcomes. They also created opportunity to conduct interviews with teachers, students, school administrators and 
families focusing mainly on their perceptions of the effectiveness, appeal and organization of the project. 

Results 

This section provides evaluations regarding seven different aspects of the program: (1) participation, (2) 
staff training and support, (3) course materials, (4) learning outcomes, (5) staff-learner interactions, (6) program 
structure and organization, (7) scale-up and future. 

Participation 

According to the quantitative data collected via online version of Form 2: Attendance, total 9440 
students completed the program. During site visits and phone calls evaluators noticed that staff had difficulty to 
find students for the program. There were two major reasons for this difficulty both of which related to the time 
of the implementations: the first, during summertime many parents preferred to send their kids to either their 
hometowns where grant parents and other relatives live or tourism places near sea shores; the second, during the 
summer time religious programs are offered to the same age group and according to the staff as well as local 
center administrators a great number of parents chosen to send their kids to these courses rather than Intel Learn. 
The staff and administrators also mentioned that the reason of this preference had nothing to do with the program 
but parents’ perception that kids will anyway learn these skills (promised by the Intel Learn) in their classes 
during the school time but they cannot learn their religion and its rules in schools. Another important point 
evaluators found out during our visits that allowing students play online games in the class during the breaks is 
an important motive to improve the participation. Some students were looking forward to play games but at the 
same time they were aware that they had to complete their activities and project in order to play games. This 
motive helped them acquire the skills the Intel Learn offered. Finally evaluators would like to state that the 
program attracts an important attention from students all over the country but its timing should be arranged better 
to be able to reach more students. 

Staff Trainings and Support 

Evaluators’ observations, interviews, and the results of Form 1 (End of Staff Training Survey) and 
Form 4 (Final Survey) show that staff trainings are big success. Up to now 362 teachers have been trained in the 
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program. The trainers got quite an experience so that they are able to handle any situation happens during 
trainings. As usual, the staff feels a little bit confused during the first day and the following but later they 
appreciate the effort put into this training and try to get more insight about the program. On the other hand 
supporting trained staff after the training was problematic in previous years. The majority of staff left alone with 
the students due to organizational problems. Although it was not enough, the only support they could find was 
from other staff. However, MEB along with the Intel project coordinator developed a new model and had a new 
agreement concerning flow of information and support. So, now the staff are be able to get enough support not 
only from trainers and MEB representatives but also from the project coordinator. According to this model there 
is a local MEB representative who is coordinating the information flow between all parties. On the other hand, 
evaluators still think that an online community of practice approach should be employed to be able to provide 
efficient and effective support. An online portal designed and developed by volunteer staff and supported by 
MEB, Intel representatives, the project coordinators and evaluators might help establishing a better interaction 
among staff and other parties, and information and experience sharing. This environment will work better if the 
staff feel comfortable in it; therefore, it should be created either by themselves or by Intel. 

Course materials 

Evaluators’ observations, interviews, and the results of Form 4 (Final Survey) reveal that course 
materials are well designed for the program. The staff as well as students indicated that they found all the 
required information in their books. However, skills book is a little bit hard to handle especially for younger 
students. An online version of it might work better. Additionally, in previous observations the staff were 
complaining about inappropriateness of some of the activities but during the last visits evaluators did not receive 
any such complains. Also, the staff stated that those parents who saw especially the skills book expressed their 
appreciation and wish to get a copy of these materials to be able to use at home. 

Learning Outcomes 

Each member of the evaluation team strongly agreed that the Intel Learn definitely help learners acquire 
the technology, collaboration and critical thinking skills but in different ratios. The program helps students 
develop technology skills. However, despite the developments, students are not able to develop collaboration and 
critical thinking skills in a satisfactory level. One of the main reasons of this problem was that these skills were 
not promoted before and unfortunately are still not in some classes even though new MEB curriculum focuses on 
these skills. Since staff and students were not accustomed to problem based collaborative learning, some of them 
had difficulty to adapt this approach required in Intel Learn Program which in turn affected the outcomes. On the 
other hand, evaluators’ observations showed that after first and second batches, staff were getting more used to 
this approach of teaching and implementing the program as it is planned. Students of these staff also 
demonstrated better collaboration skills and creative thinking examples during evaluators’ observations. In an 
interview with one of the staff she mentioned that she has a daughter a few years younger than her students in the 
program. Her daughter joined her class everyday because she did not have any place to leave her during summer. 
One day at home she told her daughter to do her homework and her daughter told her that “. . . why I do not have 
any partner. It is so easy to do the job when you have a partner and I can do better homework if I had one”. The 
staff was shocked and asked her where she learnt all these. Her daughter’s answer shocked her more: “I know 
because sister Ceylin [the names have changed] was working alone during the beginning of the school [the 
program batch] and her works were very bad although she had good ideas. But later she started to work with 
Selim who was better with computers and after that their works were the best.” So, this case is good evidence 
that the program help learners gain the required skills if it is implemented as it is foreseeing. Evaluators all 
agreed that experience made a difference in learning outcomes of the program. In sum, students took better 
technology skills, collaboration and problem solving skills with them from the program. 

Student and Staff Interactions 

Same as learning outcomes, although staff bring their past experiences and habits into the program 
implementations, the experience help them get better in guiding students rather than teaching. Evaluators 
observed positive and negative examples of the staff- students interactions. For instance, in a center, 
unexpectedly a young staff, after his student asked, started to show how to change the color of a clipart on the 
computer rather than directing his student to skills book or other students. Later during the interview he indicated 
that the best way to learn a skill was watching the correct demonstration first. Evaluators tried to convince him 
about the exploratory learning philosophy of the program but evaluators could not succeed it. Also in another 
center, evaluators came across a chaotic situation in the class and the staff was not there. Evaluators found him in 
another room talking with other teachers. Evaluators noticed that he was not really motivated about the program 



12 




and so he was trying to fulfill the requirements of the program but had no intention to encourage students to 
explore and discover. He was leaving students alone and providing almost no guiding. He was an 
older/experienced teacher. During the interview he complained us about the money they got for this project and 
expressed that the project actually does not bring anything new for them and they have already been implying 
the same (problem based collaborative learning) approach in his classes for years. On the other hand, evaluators 
met another staff via emails. He was quite old as a public school teacher but was very enthusiastic about the new 
role he had to play in the class and looked forward for the opportunities to encourage students to ask questions 
and seek for their answers. He wrote us that he was giving a small prize such as a candy bar to the student who 
asked the best and most useful question. He also mentioned that he was thinking about retirement before the staff 
training. In fact he went to staff training to relax and spend several days out of his daily duties and environment. 
But starting from the training he felt the difference of the program that helped him remember something he lost 
years ago: the joy of helping students learn. Now, he is not thinking of retirement. Similarly in a center located 
underdeveloped part of Istanbul both of the staff were having hard time to ignore the students’ request for direct 
demonstration of the tasks they are supposed to do. One of them had to yell students and told students that the 
first student who come her to ask direct demonstration of a task before going to her/his friends and/or checking 
the skills book was going to get the penalty of standing still at the comer of the class for at least 15 minutes. She 
actually did not apply this penalty to any students. In the light of these observations and more evaluators 
conclude that staff even with a few instinct motivation performs better than those who do not have any even 
though you try to provide as much as you can to motive. So that during the staff selection the project coordinator 
and MEB must be more careful. 

Program Structure and Organization 

In terms of instmctional structure and organization, evaluators have not came across any problem and 
almost all the staff, MEB representatives and students find the program successful. However evaluators noticed 
several problems in terms of project management especially first half of this year. Almost all these problems 
related to the bureaucracy. For instance it took quite a while to find a way to pay some money to the staff 
involved in the program. Another problem that affected our data collection efforts was the information flow 
especially between MEB and the staff. MEB could not establish a healthy system to learn which staff was 
implementing the program when. But after attaining local MEB representatives, all the parties including 
evaluators are getting timely and enough information. In addition organizing all the staff trainings in one center, 
Ankara was a very appropriate decision. All the staff had chance to meet with MEB upper level administrators 
and Intel representatives. This helped them to understand the importance of the program for MEB. Evaluators 
definitely recommend organizing all the future trainings in Ankara if it is possible. 

Scale-up and Future 

MEB representatives and the project coordinator mentioned that some of the staff who joined the staff 
trainings will work as trainers and offer staff training in local provinces in 2007. Although this organization 
might help to train more staff, evaluators are not sure how effective those trainings would be. But of course this 
is just a feeling and this organization might work fine. Evaluators think that an online or blended version of staff 
training might be better. Also, establishment of an online environment where all the staff can came together and 
share their experiences and feelings might help to develop the sense of community and connectedness among 
staff. This might increase their commitment to the program and in turn might affect their performance positively. 
This sort of an environment also may help us establish a better interaction and get better insight about their 
feelings. This environment should also work for presenting the students works. 

Additionally, mostly through the enthusiasm of some of the staff such as Ali Bilgi news about the 
implementation of the program take place in local newspapers. Especially, one news had a very impressive title: 
“They are still in classroom but never complain never bored”. Evaluators think that these kinds of news along 
with MEB representatives’ positive attitudes toward the program create a positive impression about the program 
and provide public support and recognition. 



Conclusion 

The results are quite promising. Although bureaucratic procedures, teachers’ old habits, preconceptions 
and students lack of previous experiences have caused some problems concerning implementation of the project, 
the Intel® Learn helped students acquire and/or improve technology, collaboration and critical thinking skills. 
Both quantitative and qualitative data showed that students’ acquisition of the technology skills exceeded the 
expected level and that they gained collaboration skills around the expected level. However, the staff should 
work more to improve students’ critical thinking skills. 
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Hybrid classes with flexible participation options - If you build it, how will 

they come? 



Brian J. Beatty 

San Francisco State University 

Abstract 

This presentation reports on the participation patterns observed in four graduate courses offered at a large, 
urban, public university in 2006-2007. All courses were taught by the same instructor. This instructor has been using 
hybrid teaching methods for more than a decade at several levels of public education, and recently developed a 
hybrid course design encouraging flexible student participation patterns - the HyFlex course. All students in this 
study were enrolled in a graduate program in Instructional Technologies leading to a Master of Arts degree. In each 
course, a mix of face-to-face and online students used a course website (hosted in an open source Learning 
Management System) to share files, access course information, review past class discussions in various formats, and 
engage in occasional topical discussions. In addition, online students had the option to participate in live online 
sessions using a synchronous web conferencing tool. All students were invited to participate either in face-to-face 
sessions or through online activities in any given week of the semester, depending on their needs and desires. 

Student participation mode (in-class or online) did vary considerably from week to week in each course. Most 
students reported that they valued in-class activities and static website resources more than synchronous online 
sessions or multimedia archives of synchronous (in-class or online) activities. Students felt a strong connection to 
the course instructor, and most students reported that they met or exceeded their learning expectations. The paper 
includes a sample of student comments regarding the HyFlex course experience, with a link to raw (aggregate) 
survey data. 



Introduction 

The kaleidoscope of teaching and learning continues to shift as instructional technologies evolve and 
revolutionize the instructional landscape. Currently, the growth in the use of online technologies to support and 
grow communities - social networks of people with a common interest - has led to an increased use of social 
networking technologies in education and training. In addition, the rush to develop online-only courses and 
programs in public U.S. universities seems to have abated somewhat, as the economic realities of limited online 
enrollments, expensive course development costs, and constricting state budgets for public education have replaced 
promises of masses of distant students paying dearly to attend [fill in the blank] ’s latest degree program (Zemske 
and Massy, 2004). The “HyFlex” course design was developed to meet the needs of a graduate program to attract 
and serve distance learners without creating online-only course sections or stand-alone e-leaming substitutes. 
(Beatty, 2006) 

What is HyFlex? 

Hybrid - combines both online and face-to-face teaching and learning activities 

Flexible - students may choose whether or not to attend face-to-face sessions . . . with no 

“learning deficit” 

The HyFlex course design was developed through a formative research process (Reigeluth and Frick, 1999) 
and introduced in 2005-2006 to meet the needs of the Instructional Technologies program at SF State to include 
online students in courses being taught in on-campus classrooms. In HyFlex courses, a mix of face-to-face and 
online students learn together as they use a course website to share files, access course information, review past 
class discussions in various formats, and engage in occasional topical discussions. (See table 1.) In addition, online 
students may have the option to participate in live online sessions using a synchronous web conferencing tool. All 
students are invited to participate either in face-to-face sessions or through online activities in any given week of the 
semester, depending on their needs and desires. 

The Learning Management System (LMS) in use at this university, iLearn (a Moodle derivative), enables a 
dynamic teaching and learning setting where the artifacts from the learning activities of face-to-face students (such 
as audio recordings) can become “learning objects” for online students, and the artifacts of online students (such as 
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discussion forum posts and chat transcripts) can likewise become learning objects for face-to-face students. (See 
figure 1 .) Because of the seamless nature of this sharing of course content, resources, and even some activities, there 
is the potential for students to cross over from one participation mode to the other (and even back again) without a 
major disruption in the “flow” of their learning processes during the semester. (Note: The HyFlex courses used in 
this study were developed for in-class teaching setting, but are under constant evolution as the instructor learns 
more about hybrid teaching and learning modes.) Since this way of teaching requires extra instructor preparation 
and involvement throughout a teaching week, an important question is, (how) do students benefit from this flexible 
approach to participation? 

Table 1. Comparison of in-class and online participation for a sample week in ITEC 801 course 



ITEC 801 Week 2: Needs and Performance Analysis 



Agenda for in-class participation: 

1. History of ISD - Where did it come from? (30 
min) 

Whole class discussion of Reiser article 

2. A Brief Overview of Instructional Systems 
Design - the process perspective (30 min) 

Review of Chapter 1, recap of Week 1 discussion 

Break - peruse sample final projects from 
archives 

3. Needs Analysis Theory (Chapter 2) (45 min) 
Performance Analysis 

Needs Assessment 
Job Analysis 
Instructional Goals 

4. Choosing a design project topics - discussion 
(30 min) 

EXTRA: Try the quiz! "Assessing Performance" 
Remember, the quiz is designed to help you 
decide how well you are learning and 
remembering some of the main concepts your text 
and our class discussions have been covering, 
(quizzes are optional, not graded, and can be 
taken as many times as you like) 



Agenda for online participation: 

1 . Read chapter 1 in text - A Brief Overview of 
Instructional Systems Design 

2. Read chapter 2 in text on Needs Analysis Theory, 
As you read, consider the context for your design 
project. (Review Week 2 presentation slides/notes 
posted to site.) 

A. Post to the "Needs Analysis Components" 
discussion 

B. Post to the "Instructional Goals" discussion 

3. Review sample design project reports - posted 
online in the "Sample Design Reports" folder. 
(These samples are located in the Week 3 space in 
iLeam.) 

4. Post your ideas for a design project to the 
"Project Topics" forum. (This forum is located in 
the Week 3 space in iLeam.) 

5. Post your weekly reflection. 

EXTRA: Try the quiz! "Assessing Performance" 
Remember, the quiz is designed to help you decide 
how well you are learning and remembering some 
of the main concepts your text and our class 
discussions have been covering, (quizzes are 
optional, not graded, and can be taken as many 
times as you like) 



For next week (this is the same for all students): 

Read Textbook: Chapter 5 - Learner and Context Analysis 
Assignment Due: Reflection post (iLeam) 
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Figure 1. Screenshot from SF State’s iLearn LMS of a sample week in ITEC 801 course 



2 □ 
Session 2: Needs and Performance Analysis, Human Performance Technology 

Quizzes far R&vi&w and Fun - *** optional 

[7l ISD Foundations 
[Tl Analyzing Performance 

If you attend class in-person: 

Week 2 Agenda (in-class session) 

If you attend class online {you do not come to class meeting 1 !: 

g| Week 2 Agenda (for online participants) 

istorical Perspectives 

Needs Analysis Components 
^ nstructional Goals 

Resources for in-class AND online activities: 

^ Discussion slides from class: History of ID (class notes included) 

2l Audio archive: Discussion History of ID; Session 2: Fall 2007 (,m4afile - plays in 
QuickTime/iTunes) 

^ Eliminate Livet Class Session from Feb 5, 2007 
^ Analyzing Performance and Assessing the Need for Training (Week 2) 

2L Audio archive: Discussion P erform an ceV Needs Analysis; Session 2: Fall 2007 (,m4a file - plays in 
QuickTime/iTunes) 

sQ Performance Analysis and Need Assessment Presentation movie (slides +■ audio) 

Theory into Practice Database (TIP) 

Human Performance Technology Resources and Links 



Return to top | wkl | wk2 | wk3 | wk4 | wk5 | wk6 
wk7 | wk& | wk9 j wklO | wkll | wk12 | wkl 3 | wk14 | wkl 5 



HyFlex Course Design Principles 

The HyFlex course design is built around four fundamental principles: Learner Choice, Equivalency, 
Reusability, and Accessibility. I believe that these principles are important in all instances of effective HyFlex 
course implementations, and may be considered “universal” principles (Reigeluth, 1999). 

Learner Choice: Provide meaningful alternative participation modes and enable students to choose 
between participation modes weekly (or topically). The primary reason a HyFlex course design should be 
considered is to give students a choice in how they complete course activities in any given week (or topic). Without 
meaningful choice, there is no flexibility . . . and therefore no HyFlex. This requires that an instructor value 
providing participation choice to students more than s/he values forcing everyone into the “best” way of learning a 
set of content. 

Equivalency: Provide equivalent learning activities in all participation modes. All alternative participation 
modes should lead to equivalent learning. Providing an alternative approach to students which leads to inferior 
learning “by design” is poor instructional practice and is probably unethical. Equivalency does not imply equality, 
however. An online learning experience (i.e., asynchronous discussion) may turn out to be much less socially 
interactive than a classroom based discussion activity. In each case, however, students should be challenged to 
reflect upon learning content, contribute their developing ideas to the discussion, and interact with the ideas of their 
peers. 

Reusability: Utilize artifacts from learning activities in each participation mode as “learning objects” for 
all students. Many class activities which take place in classrooms can be captured and represented in an online- 
delivered form for online students. Podcasts, video recordings, discussion transcripts or notes, presentation files and 
handouts, and other forms of representation of in-class activities can be very useful - both for online students and 
for classroom students wishing to review after the class session is finished. In a similar way, the activities completed 
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by online students, such as chats, asynchronous discussions, file posting and peer review, etc. can become 
meaningful learning supports for in-class students as well as provide useful review materials for online students. 

And indeed, artifacts from some learning activities, such as, glossary entries, bibliographic resource collections, and 
topical research papers, could become perpetual learning resources for all students in future courses as well. 

Accessibility: Equip students with technology skills and access to all participation modes. Clearly, 
alternative participation modes are not valid alternatives if students cannot effectively participate in class activities 
in one or more modes. If a student is not physically capable of attending class, then in-class participation is not an 
option for that student. If a student does not have convenient and reliable Internet access, then online participation 
may not be a realistic option for that student. Students need the technologies (hardware, software, networks) and 
skills in using technology in order to make legitimate choices about participation modes. It may be incumbent upon 
an instructor or academic program to provide resources and extra training to students (and instructors) so that 
flexible participation is a real option. 

Another key aspect of accessibility is the need to make all course materials and activities accessible to and 
usable for all students. For example, audio or video recordings should include text transcripts or be closed captioned, 
web pages and learning management systems must be “screen reader friendly”, and all forms of online discussion 
should meet universal design guidelines for accessibility. As more students with varied learning-mode abilities enter 
graduate programs and public, regulatory and legal pressures for universal design for accessibility increase, this 
aspect becomes increasingly important. 

Study Questions 

The questions this study attempts to answer for the HyFlex course delivery context are, When given the 
option to choose between online and face-to-face participation modes, which method do students choose? Why do 
they choose one method over another? When do they change their participation pattern, and why? Do their 
participation mode preferences change over the course of a semester? Are students satisfied with their interactions 
and learning in a HyFlex course? Answering these questions will help faculty choose a more effective mix of 
instructional options (overall class participation modes, online and face-to-face activities) to meet the needs and 
desires of their students. 



Method 

Course participation data was gathered during the Spring 2007 semester in four Instructional Technologies 
courses taught by the author. This data is either typical attendance information (who came to class, who did not, 
etc.), or evidence of online participation captured through the iFeam Teaming Management System, usually 
measured by discussion fomm participation. Online participation data includes participation frequency and quantity. 
One end-of-semester survey was used to assess student participation intentions and perceptions. Completing the 
survey was voluntary. The survey was available for one week after the last in-class meeting of the semester. Thirty 
four of 44 students (four students were enrolled in at least two of these classes) completed the survey, including 1 3 
of the 15 students registered in the “online-only” section of ITEC 801, 801.02. Table 2 lists the questions asked in 
the survey. 

For the participation part of this study, the analysis consisted of counting the frequency and amount of 
student participation online, and counting the frequency of class attendance. This data was entered into a 
spreadsheet for comparison and graphical analysis. Table 3 (in Results section) summarizes course participation for 
all four courses. 

For the student satisfaction part of this study, survey results were tabulated and entered into a spreadsheet 
for comparison and graphical analysis. This analysis produced general answers to questions about student rationale 
for participation choices, changes in participation mode or amount, and student satisfaction with various components 
used in each course. (Note: Rigorous statistical analysis of survey results was not attempted, but is planned for 
future surveys. The survey instmment with tabulated aggregate data can be accessed at the HyFlex Course Design 
website: http://itec.sfsu.edu/hyflex/hyflex home.htm ) 
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Table 2. End-of semester survey questions (Note: answer format has been changed to conserve space) 



1. Please select which course(s) you were enrolled in during Spring 2007. 

ITEC 801.01 (face-to-face: Monday night); ITEC 801.02 (online section: Wednesday night); ITEC 850; ITEC 
865 

2. Which mode of participation did you plan on using during the semester? 
face-to-face only; online only; a mix of online and face-to-face 

3. How far is your commute to SF State? 0-5 miles; 6-20 miles; 21-50 miles; greater than 50 miles; Other: 

4. On a typical class day, how long does your commute to class take? Please include time to find parking and 
walk to class. 

0-30 minutes; 31-60 minutes (one hour); 61-90 minutes; 91-120 minutes (two hours); more than two hours; 
Other: 

5. How many OTHER HyFlex courses have you taken in the ITEC program? Do not include classes completed 
this semester. 

0 (none); 1 (one); 2 (two); 3 (three); 4 (four) 

6. Please rate the value of each type of learning resource used in this class to your learning experience. What was 
valuable to you? What was of little or no value? {Note: each element is rated from Very High to Very Low) 
Face-to-face discussions; Face-to-face presentations; Online asynchronous discussions; Archived discussions 
from class meetings; Online synchronous sessions; Archived Eluminate Live! Sessions; Website links; Linked 
readings 

7. Please indicate how "connected" you felt to the following during class this semester. Compare this class 
experience to other recent class experiences. {Note: each element is rated from Very Strong to Very Weak) 

Peers; Instructor; ITEC Program; SF State 

8. 1 feel that I learned as much or more than I expected to in this class. 

Strongly Disagree; Disagree; Undecided; Agree; Strongly Agree 

9. 1 wish all of my courses were available (pick one): 

fully online; complete in-person; blended (instructor decides what is online and what is face-to-face); HyFlex 
(hybrid with flexible participation); on CD-ROM for independent study; Other: 

10. Please comment on your participation throughout the semester. Did you vary your mode from face-to-face to 
online or vice versa? If so, why? Did you prefer one mode over the other? If so, why? 



Results 

Part A of the Results section reports student participation patterns: aggregate for all courses, broken out by 
course and by individual student. Part B of the Results section reports selected findings from the end-of-semester 
student survey of participation and satisfaction with their HyFlex experience. 

Part A: Student Participation Patterns 

In Spring 2007, the author taught four courses, ITEC 801.01 and 801.02 (two sections of the same course, 
Instructional Systems Design), ITEC 850.01 (Design and Management of Training Projects), and ITEC 865.01 (E- 
Learning Development). Of the four courses, ITEC 801.01 and 801.02 were identical except that 801.02 was listed 
in the course catalog as an online course. Students were alerted ahead of time that even if they registered for ITEC 
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801.02, they would still be welcome to attend 801.01 in person if they desired. Table 3 shows summary data for 
overall average weekly participation mode for each of the four courses. 

Table 3. Spring 2007 Student Participation Mode Summary 



Course 


Enrollment 

(N) 


Attendance (weekly ave STD over the semester) 
in-class online absent 


801.01 (in-class) 


15 


7.20 | 4.69 


5.07 | 4.01 


2.67 | 1.50 


801.02 (online) 


13 


1.13 | 1.41 


8.93 | 1.98 


2.87 | 1.68 


850.01 


10 


2.38 | 1.98 


1.93 | 2.46 


0.21 | 0.43 


865.01 


10 


7.85 | 2.85 


1.15 | 2.03 


1.00 | 1.35 



Looking at the summary data, it is clear that in all courses, students took advantage of the HyFlex flexible 
participation option, with several students in each course using an alternative option (online for an in-class course, or 
in-class for an online course) each week. Not surprisingly, the greatest variation in participation mode was for the 
801.01 course since there were both in-class and online course sections for 801. In 850 and 865, participating online 
often meant being the only online student, or being one of only two or three students interacting online in any 
particular week. 

Individual Student Participation 

In 801.01, the in-class section of the ISD course, there was at least one student participating online each 
week. Two class sessions were conducted only online, in part to ensure students had the basic skills to survive in the 
online participation mode. All but one student chose to participate online at least one more time during the semester. 

In 801.02, six of 13 students never came to class in person during the semester. More than half of the 
students (seven of 13) exercised the option of coming to class to participate in-person at least once, but none more 
than 5 times (one student came to class 5 times, one came 4 times, one came 3 times, one came 2 times, and 3 came 
only once). 

In 850.01, one week of the course was conducted completely online. All but two students (80%) chose at 
least one more week to participate online during the semester. Three students (30%) chose online three additional 
times (beyond the required one online week), three (30%) chose online two additional times, and two (20%) chose 
online one additional time. 

In 865.01, students were never required to participate online during the semester, and three students (30%) 
came to class every week. Two students (20%) chose online three times, three (30%) chose online twice, and one 
chose online once. 

Part B: Student Survey of Participation and Satisfaction 

The end-of- semester survey asked students about their participation preferences, details about their 
commuting time and distance from campus, satisfaction with their learning, with the HyFlex course design, and their 
feelings of “connectedness” to various components of the learning community, and their valuing of various 
instructional resources used in the course. In this paper, I summarize the findings related to instructional resources, 
feelings of connectedness, satisfaction with learning, preference for course design, and overall satisfaction with their 
HyFlex participation experience. 

Item 6 of the survey asked students to evaluate eight types of instructional resources provided in each class. 
(See table 4.) The four elements valued the most (ranked very high or high) from students were face-to-face 
discussions (23 of 34), presentations (26 of 34), website links (26 of 34), and linked readings (30 of 34). Essentially, 
these are in-class resources and online “expert” non-interactive resources. The interactive online resources were 
valued by less than 50% of the students, in comparison. 
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Table 4. Selected Student Survey Results 



6. Please rate the value of each type of learning resource used in this class to your learning experience. What 


was valuable to you? What was of little or no value? 














Number of Responses 






Very High 


High 


Neutral 


Low 


Very Low 








(so-so) 




(or did not access) 


Face-to-face discussions 


17 


6 


5 


0 


6 




Face-to-face presentations 


16 


10 


3 


0 


5 




Online asynchronous discussions 


8 


9 


14 


3 


0 




Archived discussions from class meetings 


5 


10 


11 


4 


4 




Online synchronous sessions 


6 


16 


6 


2 


2 




Archived Elluminate Live! Sessions 


4 


4 


10 


3 


12 




Website links 


15 


11 


7 


1 


0 




Linked readings 


14 


16 


4 


0 


0 




Totals 


85 


82 


60 


13 


29 




7. Please indicate how ’’connected” you felt to the following during class this semester. Compare this class 


experience to other recent class experiences. 












very strong 


strong neutral 


weak 


very 


weak 




Peers 6 


14 


8 


6 


0 






Instructor 10 


19 


5 


0 


0 






ITEC Program 3 


12 


11 


4 


4 






SF State 1 


10 


10 


8 


5 






Totals 20 


55 


34 


18 


9 






8. 1 feel that I learned as much or more than I expected to in this class. 








Number of Responses 


Response Ratio 










Strongly Disagree 3 




8.82% 










Disagree 0 




0.00% 










Undecided 4 


11.76% 










Agree 17 


50.00% 










Strongly Agree 10 


29.41% 










9. 1 wish all of my courses were available (pick one): 
















Number of Responses 


Response Ratio 


fully online 








1 




2.94% 


complete in-person 








4 




11.76% 


blended (instructor decides what is online and what is face-to-face) 


8 




23.53% 


HyFlex (hybrid with flexible participation) 






20 




58.82% 


on CD-ROM for independent study 








0 




0.00% 


Other** 

** UNDECIDED 








1 




2.94% 



Item 7 asked students to evaluate their feelings of connectedness to their peers, the instructor, the ITEC 
program, and SF State University. (See table 4.) Connecting to others is often a particular challenge in online 
courses, and a lack of feeling connected contributes to attrition in online courses and programs ( Kemp , 2002 ; Rovai , 
2002) . Blended courses which implement a mix of in-class and online instruction may improve the sense of 
community (Rovai and Jordan, 2004), leading to feelings of greater connectedness. In the end-of-semester survey, 
students reported feeling strongly (marking “very strong” or “strong”) connected to the instructor (29 of 34) and to 
peers (20 of 34), but did not report feeling strongly connected to either the ITEC program or SF State in general. 

Item 8 asked students whether or not they agreed with a statement about learning as much as they expected, 
and 80% (27 of 34) reported that they agreed or strongly agreed. 
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Item 9 asked students what type of course delivery they preferred. A majority of students (59%) preferred 
the HyFlex course, many preferred another form of hybrid course, instructor-controlled blended (24%), and most of 
the rest preferred traditional classroom instruction (12%). Only one student preferred online-only course delivery. 
This result indicates that these students value the face-to-face components of traditional classroom instruction, 
perhaps the social connection they experience with peers and the instructor, and the immediacy of rich interaction 
that is enabled in the classroom but is difficult to achieve online. But what did they say about the HyFlex course 
experience, when they were free to use their own words? Some of those comments are presented next. 

Student Comments about HyFlex 

The final survey question asked students to comment on their participation over the semester. Most 
comments expressed satisfaction with the flexible participation options; many students mentioned the aspects of 
each mode of class that they especially liked or disliked. A sample of students comments (as submitted in the 
survey) is presented here. (Full comments can be viewed at the HyFlex design website.) 

I did online mostly and caught the tail end of class when I could. I missed seeing and 
interacting with my peers face to face, online was a great deal of work and I found I 
needed to really manage my time. Office hour participation , listening to archived class, 
reading other student forums, posting my own forums, responding with coherence, 
spelling grammar... wow. office hours time was made to seem optional. I felt it was key 
and would have liked to see more people there and used more robustly to share projects, 
get feedback and have a class. It was good to get to know the program. Sharing papers 
with people online for feedback didn’t work so much, people didn't know what to give for 
feedback. I can't believe how much I have learned. 

I enjoyed the hyflex mode of learning based on my very demanding schedule for work. 

As a consultant, I travel almost 4-5 days per week, and having the ability to attend or not 
attend class was very helpful. 

With a hectic work schedule, having the option to "attend" class online was ideal. The 
flexible structure kept me from dropping the course. 

I took two courses in this mode this semester, and my participation varied more in one 
than the other. Within a large class, the HyFlex approach is really useful because there is 
the sense that there will be a community to participate with in either modality. Therefore, 

I switched from online to face-to-face pretty regularly. Often, I made my choice based on 
what was going on for me that week (was I prepared for face-to-face session early in the 
week?) or the content of that week's lesson (am I going to fully understand this topic on 
my own?). I felt more comfortable participating online when I was confident about my 
knowledge for the week, and was just looking to peers to further develop that 
understanding. In contrast, in a small class, I was reluctant to participate online because I 
was unsure whether or not there would be someone to participate with, or enough of a 
dialogue to make the experience valuable. I only participated online once, outside of 
when the whole class was online, and it wasn't a dramatically different experience than if 
I had just skipped class for the week. I wouldn't recommend the HyFlex course for small 
classes, unless there is a more structured activity aside from discussion that one could do 
for the week. 

My participation has been about the same with online and in-class. This was my first time 
having (the option of) an online class. Being online gave me more flexibility with my 
work schedule, but I still needed to have some face-to-face peer work and discussions, 
especially regarding the materials we read and talked about. I don't think I think that i 
prefer being in class. I benefit more with being engaged, or even just listening to a 
discussion, but the availability of doing some of the work online really helped me. 
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Throughout most of the comments, students expressed an overall preference for traditional in-class 
meetings, especially the social nature of the seminar classroom. However, they also often expressed appreciation for 
the flexible nature of participation so that their studies could more easily integrate into the rest of their busy lives. 

Conclusions 

If you build it (the online options for traditional classroom-based courses) they will come! At least many of 
these students did. Approximately 15% of the student participation instances (96 of 642 opportunities to choose) 
took place in alternative modes, primarily in-class students participating online. 

Students who register for an online course may not be able to attend class in-person, even if that is an 
option. If they have chosen an online course section instead of an in-class section, it is probably for as good reason, 
such as, work conflict, family duties, tie conflicts with other classes, or travel conflicts. Based on the participation 
patterns in the ITEC program at SF State, it doesn’t make sense to prepare in-class options for students if a course is 
being offered online-only already. In this situation, if you build it, they probably won ’t come! 

HyFlex options may be more effective for students choosing online participation if there is a large group of 
students who are likely to also be online in a given week. If online participation is considered to be a “second 
choice” option (perhaps because it can be a lonely virtual space when every one else is in class), online discussions 
may be sparse and relatively ineffective as interactive discussions. 

Even when students did not choose alternative participation modes often, many report being very satisfied 
and pleased that they had the choice to participate in a different mode if they had to (or wanted to). Almost all of the 
students surveyed preferred a mix of online and classroom participation modes, and most of these students report 
that they would prefer all of their courses adopt a student-controlled flexible approach to participation. 

Looking Ahead (with Plenty of Questions!) 

As with any complex learning setting, initial development and research leads to many more questions than 
a simple study can answer. The end-of-semester survey is being revised and will be used with Fall 2007 and Spring 
2008 courses using the HyFlex course model in the ITEC program. There are other areas to pursue, many more 
questions to ask (and eventually answer, one would hope.) Here are a few under consideration: 

Student learning: Do students who choose to participate predominantly online achieve learning objectives? 
How do their final performance products (design plans, instructional packages, etc.) compare to those of their peers 
who participate in-class 

Online learning environment: Is it possible create a rich, interactive online environment when only a few 
students are participating online in a given week? Would requiring all students, regardless of participation mode, to 
participate in online forums make the online participation richer? Could that be accomplished without increasing the 
workload for in-class students unfairly? 

Faculty workload: It isn’t fair to the faculty teaching a HyFlex course to require traditional in-class 
teaching and add on the load of facilitating online students as well. How much additional work is required to 
conduct a HyFlex course? How can the faculty load for facilitating online learners be lessened? Should faculty be 
assigned fewer students for HyFlex courses or fewer courses to teach? Can teaching assistants provide the additional 
instructional support needed? 

Student outcomes: All of the students in this study are enrolled in the Instructional Technologies (ITEC) 
graduate program at SF State. Since these students are learning how to design instruction for a wide variety of 
education and training settings using technology, their experiences using online technologies in the HyFlex course 
may affect the way they think about teaching and learning in the various instructional contexts they move into. Does 
the HyFlex experience lead them to consider the impact of learner control and designing for flexible alternative 
approaches to learning the same content? Ultimately, do they become more effective designers and educators? 

The results of continuing studies and development of HyFlex courses at SF State will try to answer these 
questions over the next few years. At SF State, especially in the ITEC program, we intend to determine how to select 
additional existing courses for hybridization with the HyFlex model, develop new courses in hybrid formats to meet 
the emerging needs of student populations, and help instructors and students better understand how new 
technologies can support a different approach to course delivery; creating a context for full and flexible participation 
in a learning community that may be more effective that traditional single-mode course designs. 
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Abstract 

Students engaged in Problem-based learning (PBL) units solve ill- structured problems in small groups, and 
then present arguments in support of their solution. However, middle school students often struggle developing 
evidence-based arguments (Krajcik et ah, 1998). Using a mixed method design, we investigated (1) the impact of the 
Connection Log-a computer-based argumentation scaffold system-on middle school students’ construction of 
evidence-based arguments during a PBL unit, and (2) scaffold use among members of two small groups 
purposefully chosen for case studies. Data sources included a test of argumentation, debate rating scores, videotaped 
class sessions, and retrospective interviews. Findings included a significant impact on individual argumentation 
skill, and use of the scaffolds by the small groups to communicate and keep organized. 

Introduction 

Recently, science educators have called for the use of inquiry-based instructional frameworks such as 
problem-based learning (PBL) to help secondary students to learn the process of science (i.e., the inquiry process), 
and move beyond instruction focused on declarative knowledge (e.g., Keys & Bryan, 2001; Sandoval & Reiser, 
2004). In PBL, students center their learning on ill-structured problems, or problems for which (1) the initial 
statement does not indicate how to represent the problem, and (2) there are multiple paths to a solution and multiple 
solutions to the problem (Hmelo- Silver, 2004; Jonassen, 2003). 

Within ill-structured problems, developing evidence-based arguments (EBAs) is essential to producing a 
viable solution (Cho & Jonassen, 2002). EBAs consist of discussions that provide support for claims with evidence 
and premises (Perelman & Olbrechts-Tyteca, 1958). However, secondary students often struggle creating EBAs 
(Krajcik et al., 1998; Sandoval & Reiser, 2004). Middle school students’ inability to construct EBAs during PBL 
could be attributed to three challenges: (1) adequately representing the central problem of the unit (Ge & Land, 

2004; Liu & Bera, 2005), (2) determining the most relevant evidence they must gather and gathering it (Pedersen & 
Liu, 2002-2003), and (3) synthesizing gathered information to construct a sound argument (Cho & Jonassen). 

Possible Solutions 

While some authors advocate direct instruction to help K-12 students improve argumentation skills 
(Knudson, 1991; Voss & Means, 1991), such interventions are not always successful (Knudson; Marttunen & 
Laurinen, 2001). Also, while direct instruction may help secondary students engaged in PBL units learn the 
components of EBAs, such students may continue to struggle representing the problem, determining relevant 
information to gather and gathering it, and synthesizing the gathered information to construct a strong argument. 

Scaffolding is support (from a teacher or computer tool) provided to students participating in a task they 
cannot complete without assistance (Hannafm, Land, & Oliver, 1999; Wood, Bruner, & Ross, 1978). Computer- 
based argumentation scaffolding has helped college (Cho & Jonassen, 2002) and middle school students (Bell, 1997) 
produce coherent arguments. However, many available computer-based science-inquiry scaffolds are linked to 
specific PBL problems (Quintana et al., 2004), and transfer to other problems or domains has not been shown. 

We designed and developed a computer-based scaffolding system, the Connection Log, to support middle 
school students’ development of EBAs during a variety of PBL units. The purpose of this paper is to describe the 
Connection Log and to address how and why middle school science students use computer-based argumentation 
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scaffolding to construct an argument while participating in a PBL unit, and the Connection Log’s impact on (a) 
individual student argumentation skills and (b) group argument quality. 



Method 

Design and Development of the Connection Log 

The first author designed the Connection Log based on guidelines that emerged from a review of the 
literature and detailed observations of middle school students engaging in PBL: (1) embed scaffolds within a system, 
(2) make students articulate their thoughts, (3) constrain the problem space, (4) consider motivation, (5) make 
scaffolds explicit for students with less prior knowledge, and (6) focus on the development of conceptual, 
procedural, and strategic computer-based scaffolds (Belland, Glazewski, & Richardson, in press). The third author 
reviewed the instructional analysis once and the storyboard twice. After revising the storyboard, the first author 
reviewed scaffold readability with a seventh-grade science teacher and several seventh-grade students. We 
simplified the language where possible, and implemented rollover definitions where necessary. 

Resources Used 

We used Macromedia Fireworks version 8 to storyboard. The html shell was created using Macromedia 
Dreamweaver version 8. The php code was written using vi (text editor for UNIX). Database architecture was 
modeled using Toad™ Data Modeler. A Sun server running php 5.1.4 and mysql 5.0.22 hosts the Connection Log. 

Evaluation 



Setting and Participants 

This setting was a laptop initiative middle school in a small, rural Midwestern community. The school had 
38 teachers and 543 students, 44% of whom received free or reduced lunches. The sample included eighty-six 
seventh-grade science students in four class sections. The teacher had four years experience facilitating PBL units. 

Data were gathered during a 2-week PBL unit on the Human Genome Project (HGP), which followed a 
teacher-led instructional unit on genetics and its role in human development. Each student team chose a unique 
stakeholder group such as doctors or religious leaders. Students needed to assume a position on the HGP based on 
their stakeholder’s perspective, outline a plan for promoting their position, and argue their position during a debate 
at the end of the unit in fictitious competition for a grant to further their position. 

Design 



We used a mixed methods approach to address our research questions. 

Qualitative method. We presented an in-depth description of how the members of two small groups worked 
together and used the scaffolds during the unit. 

We (1) videotaped each group during the unit, and transcribed verbatim all dialogue, (2) retrieved each 
group member’s response to prompts, (3) observed students from all class sections during the unit, and (4) 
conducted prompted, retrospective interviews of approximately 30 minutes each with each group. In each interview 
a unique, approximately 20-minute video containing scenes from the videotaped class sessions prompted 
participants’ recollection of how they used the Connection Log and why. 

We (1) coded the video and interview transcripts, (2) triangulated coding with observation and database 
information, (3) identified common themes, and (4) interpreted themes from the symbolic interactionism framework, 
according to which people interact with things based on the meaning they assign to the latter (Blumer, 1969). 

Quantitative method. We used a two-factor nested experiment with factor B (classroom) nested in factor A 
(scaffold or no scaffold) (Kerlinger & Lee, 2000). The teacher identified two classes as high-achieving and two as 
low-achieving. The low-achieving class means were significantly lower than those of the high-achieving classes, 

F(l, 81) = 6.93, p < .05, on a pretest of argumentation ability. Pretest scores did not differ significantly between the 
low-achieving, F(l, 81)=. 38, p=. 54, or the high-achieving, F(l, 81) = .62, p=.43, classes. We assigned one high- 
achieving and one low-achieving science class to each condition: scaffold and no scaffold. 

An argumentation measure was adapted with permission from the test used by Glassner et al. (2005). 
Students read four scenarios in which a claim was made about a particular topic and two supporting statements were 
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advanced. Students needed to indicate how well each statement supported the original claim. The pretest 
(Cronbach’s alpha=.71) was similar to the posttest (Cronbach’s alpha=.77). 

To address the impact of the Connection Log on individual argumentation skill, we used nested ANOVA. 
If nested effects were significant and ordinal, we calculated simple effects (Keppel, 1982). 

We videotaped all groups during the debate. The video was transcribed, and two raters blind to treatment 
condition assigned numerical scores for claim, evidence, and connection of claim to evidence quality for each team 
in all four periods. After raters came to consensus, agreement, measured by Cohen’s Kappa, was 0.99. 

To address the impact of the Connection Log on group argumentation quality, we used nested MANOVA. 
If effects were significant, we ran follow-up ANOVAs. If nested effects were significant and ordinal, we calculated 
simple effects (Keppel, 1982). 



Results 

Connection Log Description 



General Description 

The scaffolds are organized as six stages: 

1 . Define problem , in which students define the problem in their own words 

2. Determine needed information , in which students decide on evidence and information about the 
problem they need to find, and strategies for finding it 

3. Find needed information , in which students find and record the information they need to know 

4. Organize information , in which students organize the information they found to make it more useful 
when developing their claims and building their argument 

5. Develop claim , in which students actually develop their claims 

6. Link evidence to claim, in which students link evidence to their claims and build an argument 

Each stage is divided into 2-4 steps. See Figure 1 for a screenshot of step 1 of stage 1. Most steps have text 
boxes in which students type responses. Upon logout, all work is saved. Students start where they left off when 
logging back in. 

Figure 1. The Connection Log. 
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LOGOUT 





Initial 

Thoughts 



Compare 
with Group 
Mates 



v 



Come to 
Consensus 




Define Problem: Initial Thoughts 

By Myself 

Write down & statement of the problem in one sentence in the box below. This should include 
what is happening r information about the stakeholders, and how the problem affects the 
stakeholders. Don't worry if you think your definition isn't good. We'll take a look at it later. 

An example of a .statement about the problem in one sentence is "The ice caps are melting, 
and this causes people who live near sea level to be threatened" 



Type your problem definition here: 



NEXT 




Unique features. The Connection Log was developed for use with any PBL unit and guides students to 
articulate their thoughts. Because of group registration, students read what their group mates articulate at each stage. 
At each stage in the argument construction process (e.g., organize evidence, develop claim) students (1) articulate 
their own thoughts individually, and (2) discuss all group members’ articulated thoughts to form a group consensus. 

How Do Students Use Computer-based Argumentation Scaffolding? 



Group 1 



Robert, Erin, and Alejandra (note: all names were changed) faced different challenges during the unit. 
Robert was a leader who found the unit difficult due to his lack of prior knowledge about the unit content. He was 
also confused about the debate structure. 

Erin was often unsure of herself. On many occasions when either Alejandra or we asked Erin a question 
about the group’s strategy or how the HGP could help their stakeholder, she replied, “I don’t know, ask Robert.” She 
appeared to accept Robert’s lead, but complained that sometimes it seemed like “Robert doesn’t care.” She also was 
confused about the overall task. 

Alejandra had limited English proficiency. She struggled expressing herself and understanding what her 
teammates said, but was able to stay aware of her group mates’ thoughts by reading what they wrote in the 
Connection Log. In this quote she reflected on her silence: “I was listening. I always listen. I was listening to what 
she was saying. Because I was trying to help, but they always talk, and I was trying to listen, but Erin talks too fast.” 
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The group used the Connection Log to (1) get and stay organized, (2) aid communication, and (3) think of 
ideas. The Connection Log appeared to help the group to counter (1) confusion about the debate structure and 
required presentation parts, and (2) communication problems by allowing them to read what each other wrote. 

Group 2 



Daniel, Megan, and Claudia faced different challenges during the unit. Daniel was a soft-spoken leader 
who tended to take over tasks when required to explain them too much. He was challenged by the ill- structured 
nature of the unit. Though he understood them from the beginning, he found it challenging to apply the Connection 
Log’s prompts to his project early in the unit. However, as we explained them more, Daniel found the prompts 
easier to apply. 

Claudia was an English language learner who joined the group with three days left in the unit, and was 
assigned the task of finding pictures and information to complete the group’s poster. She found the unit more 
difficult than previous inquiry units due to difficulty finding information. Because of her late addition, she did not 
use the Connection Log extensively. 

Megan was a quiet student who also found the unit difficult because of its ill-structured nature. She often 
asked Daniel how to accomplish tasks. She noted difficulty getting started with the Connection Log, but once she 
got going, “it was pretty easy.” 

The group used the Connection Log to (1) get and stay organized, (2) serve as a reference, and (3) ensure 
inclusion of all required parts of the presentation. The Connection Log appeared to help the group counter their 
challenges by (1) structuring the unit to aid organization and strategy, (2) storing ideas for later reference, (3) 
indicating required parts of the presentation. 

What is the Impact on Individual Students ’ Argumentation Skills? 

The Connection Log had a significant main effect on individual argumentation ability (F(l, 8 2) =2.99, 
p=.09, ES=0.35) and a significant simple effect on the individual argumentation ability of low-achieving students 
(F(l, 82) =6. 07, p=.01, ES=0.61). In other words, the Connection Log significantly affected the individual 
argumentation ability of all students, but had an effect of greater magnitude on the individual argumentation ability 
of low-achieving students. 



What is the Impact on Group Argumentation Quality? 

The Connection Log did not have a significant main effect ( A=0.88 , F(3, 25)=1.16, p=.34 ), but a 
significant nested effect (A=0.66, F(6, 50)=1.95, p=.09 ) on claim, evidence, and connection of claim to evidence 
quality. Follow-up ANOVAs indicated significant nested effects of the Connection Log on claim (F(2, 27)=4. 77), 
p=.02) and connection (F(2, 27)=4.52, p=.02) scores, but simple effects were not significant. In other words, when 
considering all scores and student types, not all means were determined to be the same. When considering only 
claim or connection scores, scaffolding’s effect appeared to differ according to student type, but further tests did not 
provide evidence of a significant difference. 



Discussion 

How do Small Groups of Middle School Science Students Use Computer-based Argumentation Scaffolding to 
Construct an Argument while Participating in a Problem-based Learning Unit? 

When considered in context of what is currently known about how secondary students use scaffolding, the 
results related to how students used the Connection Log are interesting in that they add to the research base on how 
middle school students use hard scaffolds. 

The capacity of hard scaffolds to facilitate organization. As Daniel noted, the Connection Log “helped us 
put it [the presentation] together so it didn’t look confused and sloppy.” Most articles in which the coherence of 
arguments was examined indicate that college (Cho & Jonassen, 2003; Ge & Land, 2003) and secondary (Bell, 
1997; Kyza & Edelson, 2005) students who used evidence-based argumentation scaffolds produced more coherent 
arguments than students in control conditions (who did not use scaffolds). However, very few seemed to indicate 
why. Some authors noted that students using scaffolds more evidential support for claims (Cho & Jonassen) than 
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students who did not use scaffolds, but it is left to the reader to imagine what specifically about using the scaffolds 
led to the more coherent argumentation. 

A closer look at what leads to argumentation coherence may reveal a possible impetus for greater argument 
coherence. Coherence of argumentation depends on three things: a clear claim, adequate supporting evidence, and 
an established link between the evidence and the claim (Perelman & Olbrechts-Tyteca, 1958). Essential to 
successfully linking evidence and claim and delivering an argument is organization (van Eemeren et al., 2002). In 
past studies in which secondary students who used scaffolds created more coherent arguments (Bell, 1997; Kyza & 
Edelson, 2005), students in the experimental condition may have been more organized than students in the control 
conditions, and that the increased organization contributed to more coherent argumentation. However, given that the 
authors did not present data to that effect, one cannot know. 

The finding that the Connection Log helped Groups 1 and 2 stay more organized makes sense when one 
considers the poor organization of many groups in the control condition, many of whom jumped from one topic to 
another during the debate, and who did not appear to coordinate their efforts well during the unit. In my (first 
author) observations of all classes during the unit, I noticed that some groups in the control condition experienced 
difficulty staying organized. Often such groups researched a topic, researched the topic again later during the unit, 
and then researched it yet again. However, they were not looking for new or deeper information about the same 
topic, they were looking for the same type of information. For example, a group with the stakeholder perspective of 
adopted children in one of the control classes continually researched how many adopted children there were in 
Indiana and in the U.S. They asked me to help them find information on Day 3, and I provided them two links that 
they could use as well as strategies for finding further information. They continued to search for information about 
the number of adopted children in the US and in Indiana for much of the rest of the unit, usually finding the same 
sites. Part of the reason that they were stuck in a cycle of continually searching for the information may have been a 
lack of organization. They clearly did not keep track of search histories or strategies. 

That the groups that used the Connection Log appeared to be more organized than students who did not use 
the Connection Log is in line with the findings of Simons and Klein (2007), who found that students who were 
required to use scaffolds were more organized than those who were in the scaffolding optional and no scaffolding 
conditions. Greater organization as supported by scaffolds benefits not only argumentation, but also ill- structured 
problem-solving success (Jonassen, 2003). 

The capacity of hard scaffolds to promote inclusion of all required project parts. Daniel noted that the 
Connection Log helped his group ensure that they included all required parts of the presentation. This is consistent 
with the findings in the literature that students who are exposed to evidence-based argumentation scaffolds in the 
form of question prompts or video modeling determined more relevant information to gather than students in control 
groups (Pedersen & Liu, 2002-2003). 

Many articles we reviewed that did not employ control groups also indicated that scaffolds promote the 
inclusion of all required project parts (e.g., Kyza & Edelson, 2005). Daniel’s group’s efforts to find evidence to 
support their claims mirrored that of groups who were prompted by empty evidence boxes in Progress Portfolio to 
include more evidential support for their hypotheses (Kyza & Edelson). This could be because the Connection Log 
fulfilled the group’s conceptual scaffolding needs by guiding its members on what to consider (Hannafm et al., 

1999). However, few authors articulate what about the scaffolds promotes the inclusion of all required project parts. 
Especially when scaffolding was not designed expressly for use with a particular unit, what about the scaffolding 
makes students add more information in order to meet project requirements? In Daniel’s group’s case, it appeared to 
be the act of looking back at what his group had entered into the scaffolds alongside the prompts that guided his 
group as to what to find. Members of Group 2 thus appeared to be scaffolded not only by what we had written a 
priori in the scaffolds, but what they wrote in the text boxes of the scaffolds as they went through the unit. 

The Connection Log as reference and communication aid. A very interesting finding was that the 
Connection Log helped the members of Group 1 to communicate. Our literature review indicated that students might 
benefit from being required to input their responses to question prompts into a database that could be accessed later 
(Blumenfeld et al., 1996; Kyza & Edelson, 2005). We thought that in the process of articulating their thoughts, 
students would think about what they were thinking, and that by writing what they thought, the exact rendering of 
their thought would be available for their group mates to read and analyze. This effect appeared to be manifest in 
Erin’s experience with the Connection Log, as she noted benefiting from having to create a textual representation of 
what was in her head. 

However, the nature of the communication aid that the Connection Log appeared to provide to the group- 
facilitation of communication between the native-English- speaking members of the group and the ENL member and 
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continuity of the group’s functioning when a group member was temporarily absent-was of a nature that we had not 
envisioned or encountered during our literature review. First, the question of what scaffolding needs ENL students 
have and how computer-based scaffolds can be designed to fulfill those needs has to our knowledge not been 
addressed in the scaffolding literature. That Alejandra felt that she could stay aware of what her group mates were 
thinking and doing by reading what they wrote in response to the Connection Log’s prompts is interesting in that it 
suggests new areas of scaffolding that should be investigated. Though Alejandra is only one ENL student, the fact 
that she felt involved in the group as a result of reading what her group mates wrote rather than what they said 
suggests a possible new line of research into scaffolding in K-12 settings: what features of computer-based scaffolds 
best support ENL student’s efforts during PBL units. 

Second, Erin and Alejandra contended that the Connection Log helped the group function when Robert 
went to another part of the room. We found no literature in which cases of small groups working in PBL units are 
described in which one group member goes off to be alone, and the other group members describe what helped them 
continue during his/her absence. However, communication problems normally ensue when one of two people who 
are talking leaves in mid-conversation. That the Connection Log countered some of those difficulties by allowing 
Erin and Alejandra to read what Robert had written is interesting and merits further investigation. 

Group l’s members also noted that they appreciated being able to compare each other’s ideas. As Robert 
noted, many times group members would write one thing and say another. When this happened, the group members 
could hold a discussion about what they really meant in order to come to consensus. We found no articles in which 
the capacity of hard scaffolds to promote the process of comparing ideas was explored. However, Robert’s comment 
makes sense because often during the unit Erin found herself saying one thing and writing yet another. As she noted, 
“everything always sounds better in my head.” By being forced to commit her idea to writing, Erin also made her 
idea available to her group mates. Alejandra also committed ideas to group discussion through the use of the 
Connection Log. While we found that Robert was the group leader, not all group ideas originated with him. This 
finding is consistent with findings in the literature that computer-based scaffolds can provoke the discussion and 
explanation of ideas among group members, and that this interaction in turn can lead to better ideas. 

Another interesting finding is that the members of Group 2 used what they typed into the Connection Log 
as a reference. Though members of Group 1 did not specify that they used what they wrote in the Connection Log as 
a reference, they did note using what they wrote for the presentation, which indicates that they did consult their 
notes when coming up with their presentation. This finding is consistent with the finding that students in control 
conditions did not consult notes when coming up with a solution, while students using Progress Portfolio (in which 
they could record thoughts) did (Kyza & Edelson, 2005). However, often students engaged in inquiry units did not 
consult their paper-based notes when determining a solution (Blumenfeld et al., 1996) or evaluating why a solution 
was not ideal (Puntambekar & Kolodner, 2005). That the members of Groups 1 and 2 referred to their notes when 
determining a solution is promising and warrants further research. If the Connection Log can lead students to 
consider their research when arriving at a solution, it could help students engaged in PBL be more efficient and 
successful. Few would argue that research done during a PBL unit should be forgotten after it is performed, or that 
students should research the same topic multiple times to find the same information. 

What is the Impact of the Connection Log on Individual Argumentation Skills? 

Main Effect of the Connection Log on Argumentation Scores 

The main effect of the Connection Log on the individual argumentation scores is interesting for several 
reasons. First, articles examining the impact of hard argumentation scaffolds to tend to not employ the use of 
transfer tests of content knowledge or skills as response measures. To our knowledge none that involve middle 
school students use such tests. So our finding of a significant effect of the Connection Log on individual 
argumentation adds to the literature base on argumentation scaffolding for that reason. 

Second, the overall effect size of 0.35 indicated that the magnitude of the effect of the Connection Log on 
individual argumentation skill was medium-small (Cohen, 1969). It is important to note that the effect was achieved 
through use of the scaffolds during a PBL unit during an eight-day span. Students only started using the scaffolds, 
and then only for about ten minutes, on the third day of the unit. Though an effect size of 0.35 is not enormous, it 
does appear to suggest that the Connection Log can positively impact student achievement in a substantial way. 

Direct instruction has not produced consistent results in raising middle school and other students’ 
argumentation skills (Knudson, 1991; Martunnen & Laurinen, 2001). Thus, the Connection Log’s significant effect 
on individual argumentation skills is important in that it suggests that scaffolds such as the Connection Log may fill 
an important gap in helping middle school students raise their argumentation skills. One characteristic of successful 
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middle school “graduates” is that they are intellectually reflective people who can “analyze problems and issues, 
examine the component parts, and reintegrate them into either a solution or into a new way of stating the problem or 
issue” (Carnegie Council on Adolescent Development, 1989, p. 15). Another characteristic is that they “have learned 
to learn, a critically important capacity because of the rapidly changing nature of occupations and jobs” (CCAD, p. 
15). In short they should be able to solve problems and build effective arguments. 

Nested Effect of the Connection Log on Argumentation Scores 

The finding of a simple main effect of greater magnitude than the main effect of the Connection Log on the 
individual argumentation skills of lower-achieving students may indicate that certain aspects of the Connection Log 
supported greater improvement in lower-achieving students’ ability to recognize the extent to which a statement 
supports a claim. It is not clear why this may be the case as there is little literature that examines the differential 
impact of evidence-based argumentation scaffolding on different types of students. However, it may be that 
differences in the magnitude of the effect of the Connection Log resulted from students in the lower-achieving 
sections using the Connection Log in a different way than students in the higher-achieving sections. Insight into the 
different ways higher-achieving and lower-achieving students used the connection log may be gained from looking 
at the ways Group 1 and Group 2 used the Connection Log. Like Daniel’s group, Robert, Alejandra, and Erin used 
the Connection Log to get and stay organized. But they also used the Connection Log to aid communication and to 
compare ideas, while Daniel’s group did not. 

The interesting finding to discuss in terms of the impact of the Connection Log on individual 
argumentation skill is its use to compare ideas. The process of comparing ideas, especially when the ideas are about 
evidence and claims could potentially raise one’s skill in evaluating the evidential support for a claim because it is 
essentially practice performing the task. That is, as Robert, Alejandra, and Erin were comparing ideas and trying to 
come to consensus, they had to explain to each other why they thought that a particular piece of evidence supported 
or not their claim, or if their claim was valid. Daniel’s group did not claim to do this and did not appear to do this as 
much from the video evidence. Practice performing a task can vastly improve one’s ability to perform it. 
Observations indicated that many groups in the higher-achieving section of the experimental condition did not 
appear to engage in as many types of conversations of this type, but that other groups in the lower-achieving section 
of the experimental group did. 

The effect size for the treatment among lower-achieving students of .61 was medium (Cohen, 1969). Given 
the initial difficulties that the students experienced using it and the relatively short treatment length, the impact of 
the Connection Log on individual argumentation skill appears to be substantial, and should be investigated further. 
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What is the Impact of the Scaffolding on Group Argumentation Quality? 

The finding of no significant effect of Connection Log on claim, evidence, or connection scores was 
disappointing, and may be due to low power to detect significant differences. The power to detect significant effect 
of the Connection Log was 0.19, and the power to detect significant nested effect of the Connection Log was 0.35. 
Such low power makes it less likely that a researcher will detect significant differences. 

The finding could also be due to a ceiling effect. Sixteen out of 30 groups attained the maximum score of 
six on claim, nine out of 30 groups attained the maximum score of six on evidence, and 13 out of 30 groups attained 
the maximum score of six on connection. This large number of maximum scores leaves one to wonder whether some 
of those groups would have scored higher than six if the scale had been wider. 

There is a relative lack of studies in which transfer of scaffolded skills is measured. The lack of a 
significant effect of the Connection Log on group argumentation quality is consistent with the findings of Cho and 
Jonassen (2002), who did not find a significant transfer of argumentation scaffolds on problem-solving tasks. The 
finding is also partially consistent with a study in which scaffolds’ transfer to a problem-solving task, as compared 
to didactic and help conditions, was inconsistent (Pedersen & Liu, 2002-2003). 

Limitations and Suggestions for Future Studies 

In this section we discuss suggestions for future research using (1) the Connection Log with a similar unit, 
(2) the Connection Log with a different unit, and (3) other systems developed using the guidelines for the 
development of computer-based scaffolds to support evidence-based argumentation. 

Using the Connection Log with a similar unit. Future researchers should investigate how other small groups 
use the Connection Log and other scaffolding systems like the Connection Log in similar and different units. Also 
important is determining if students from similar and different student populations use the Connection Log in a 
similar manner as Groups 1 and 2 on a similar unit. We specifically chose to focus on how two small groups used 
the Connection Log so that we could gain in-depth information. However, such a strategy limits the potential 
generalizability of findings (Stake, 1978). It is likely that members of other groups would use the Connection Log in 
different ways. Would other groups use the Connection Log to compare ideas and/or to aid communication? Would 
they use it to get and stay organized, or to ensure inclusion of all required project parts? 

Future researchers should also use a wider rubric to assess students’ debate performances. The potential 
ceiling effect may have contributed to the lack of significant results in this study. A rubric that allows for a greater 
number of possible scores may allow more substantial differences in scores to emerge between groups. 

Using the Connection Log with a different unit. We designed the Connection Log such that it could be used 
in a variety of PBL units. As such, it would be important to determine if similar impacts on individual argumentation 
skill and group argumentation quality can be ascertained when the Connection Log is used in conjunction with units 
of differing content, length, and depth. For example, if the unit topic were global warming, and students needed to 
write a letter to Congress expressing their opinion about global warming and what should be done about it, would 
students use the Connection Log in a similar manner to Group 1, Group 2, or neither? Would lower-achieving 
students tend to use it in a similar manner to Group 1 , and higher-achieving students in a similar manner to Group 2? 
Or would they use the Connection Log in a different manner? 

Using the Connection Log with a different unit and with a similar unit, it would be interesting to determine 
the extent to which the system helps ENL students with limited English proficiency be involved in their group. It is 
possible that Alejandra was an isolated case and that other students with similar difficulties speaking and 
understanding spoken English would not benefit from reading what their group mates wrote in the Connection Log. 

Used with a unit of different topic or length, the Connection Log may have an impact on group 
argumentation quality. This can only be learned with further research. 

Using other systems. An interesting study would examine how students use other scaffolds developed using 
our guidelines for the development of computer-based scaffolds to support the creation of evidence-based 
argumentation. Would features inspired by the guidelines (e.g., networked connectivity), and exhibited by other 
scaffolding systems, be helpful to middle school students? Would such a scaffolding system show positive impact 
on individual argumentation skill or on group argumentation quality? 

Last, future research should examine when and how the Connection Log or other systems like it could be 
faded, as the literature base currently does not describe well the fading of hard scaffolds (Pea, 2004; Puntambekar & 
Hubscher, 2005). The guidelines for the development of argumentation scaffolds that recommended that all 
scaffolds be part of a system and that students articulate their thoughts caused me (first author) to design the 
Connection Log in such a way that fading it would have been difficult. Each student needed to type their answers to 
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prompts in order for group mates to be able to read each other’s input. One of the only existing models for the fading 
of hard scaffolds involves students individually deciding when they do not need the former any more (Puntambekar 
& Hiibscher). But if a group using the Connection Log were made up of Students A, B, and C, and Student A 
decided she did not need the scaffolds any more, but Students B and C deemed that they did, that would be a 
problem. We do not know how the Connection Log could be individually faded. However, that does not mean that it 
cannot be done. 



Conclusion 

Students will face unique challenges in the 21 st century, and to help them prepare, schools need to 
incorporate inquiry-based instructional activities such as PBL so that students engage in authentic inquiry in school 
(Brush & Saye, 2001; Gallagher, 1997; Keys & Bryan, 2001; Sandoval & Reiser, 2004). Creating evidence-based 
arguments is central to student success during a problem-based learning unit (Jonassen, 2003). Doing so is 
particularly difficult for middle school students due to challenges (a) adequately representing the problem that is the 
focus of the unit (Ge & Land, 2004; Liu & Bera, 2005), (b) determining the most relevant evidence that they must 
gather and the process of gathering it (Pedersen & Liu, 2002-2003), and (c) synthesizing the information that they 
gathered to construct a sound argument (Cho & Jonassen, 2002). 

Scaffolding has been advocated as one way to help college (Cho & Jonassen, 2002) and secondary (Bell, 
1997) students build evidence-based arguments while participating in PBL unit. We designed and developed the 
Connection Log to support the creation of evidence-based arguments among middle school students engaged in a 
variety of PBL units, and implemented it in two out of four class sections participating in a PBL unit on the HGP. 
We detected no significant differences on claim, evidence, or the connection of claim to evidence ratings of debate 
performances. However, students used the Connection Log in different ways, and we found a significant main effect 
of the Connection Log on individual student argumentation ability, as well as a significant simple main effect of the 
Connection Log on the individual argumentation ability of lower-achieving students. Results of this study show the 
potential of the Connection Log to help middle school students overcome the challenges of creating evidence-based 
arguments. 
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One Laptop for Each Middle Schooler: Practical Findings from a Recent 

Study 

Ward Mitchell Cates, Lehigh University 

Ubiquitous computer access has long been touted as making a difference in the educational endeavor (see 
Gayeski, 1989; Kay, 1984; Ravitz, 1999; Rogers, 2001). One version of such access takes the form of using 
individual wireless laptop computers (Hill, Reeves, & Heidemeier, 2004; Rockman et ah, 2004; Russell, Bebell, & 
Higgins, 2004). Such computers provide each student in a class with anytime/anywhere access to instructional 
programs, the World Wide Web, and productivity tools that may be employed for instructional purposes. In recent 
years, a number of such “one-to-one” laptop initiatives have been implemented for individual schools, entire school 
districts, and entire states (Ubiquitous Computing Evaluation Consortium. 2005). As Zucker (2004 ) and Ross., 
Lowther, and Morrison (2001) noted, however, not all implementations have been rigorously evaluated. 

This paper reports findings from a recent study of a one-laptop-per-student initiative in grades six and 
seven of a four middle schools in a single school district in the Northeastern United States. The study was conducted 
over 9 months (October through June). During this period, students had the same “personal” computer to use every 
day, although they were not able to take their computers home at night. Students used Apple iBook™ laptop 
computers to take notes, to explore a variety of Websites, to prepare word-processed documents for their various 
content subjects, to prepare presentations, to conduct research online, and for other similar group and individual 
computer activities. This study used data triangulation across multiple points in time and multiple respondents, 
including observational data to assess how the studied laptop implementation affected teachers and students, with a 
particular focus on changes in teaching/leaming practices in the classroom. While not unique (see for example, 

Bums & Polman, 2006; Metiri Group, 2006; Sargent, 2003; Silvemail & Lane, 2004, Windschitl & Sahl, 2002), this 
study was rigorous in deriving its data sources and in seeking to correlate related data sources to produce a clearer 
picture of the impact of the implementation on teacher and student attitudes and teaching/leaming skills. 

Data Collection 

In mid-October, I interviewed the principals of the four middle schools, as well as the assistant principals 
for curriculum (sixth grade). Each interview took approximately one hour and used the same interview protocol 
instmment. During the four weeks from mid-October to mid-November, all sixth and seventh-grade teachers, as well 
as affiliated special education teachers, were surveyed. The response rate for teachers was quite high; all but a 
handful completed the survey and returned it. All sixth-grade students were also surveyed during this period. In 
addition, all sixth-grade teachers were observed teaching one class. These measures were intended to supply 
baseline data. At this point not all laptops had been distributed, nor were they in frequent use in schools. 

The first week of Febmary, I interviewed all four principals and all four assistant principals for curriculum 
once again, this time using a slightly modified version of the initial interview protocol. While at each middle school, 
I left surveys for all sixth and seventh-grade teachers, as well as affiliated special education teachers, to complete. 
Within a week, I had received the vast majority of teacher surveys. Once again, the response rate was high, although 
a slightly higher percentage of blank or partially blank surveys were returned. 

In late April and early May, all 42 sixth-grade teachers were observed teaching one class. In May and early 
June, all sixth and seventh-grade teachers, as well as affiliated special education teachers, were surveyed a final 
time. In this same time period, all sixth-grade students were also surveyed. In mid- June, I interviewed the principal 
and assistant principal for curriculum at each of the four middle schools. Once again, the same protocol was used for 
all interviews, although this protocol was once again slightly modified from the midpoint version, in this case to 
focus more on summative assessments and reflections. 

This study employed classroom observations. Clearly not all instructional activities are suited to the use of 
technology and one takes the luck-of-the-draw in whether one will see technology in use during a single 50-minute 
observation. We did not ask teachers to modify their plans and use technology so that we could observe its use. We 
figured instead that observing classes, regardless of whether they were using technology during that period, would 
produce a more representative picture of how often and in what ways technology was being used in the schools. Of 
course, this decision may account for the percentage of teachers who were not using computers during each 
observation period. 
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All surveys reported herein constitute self-report data. A caution is in order: It is important not to mistake 
self-report data for factual evidence. That is, what someone reports on a survey may or may not be accurate. Self- 
report data may be affected by respondent awareness, experience, perceptions, and divergent intentions. 

Findings 

This section is divided into four subsections: Student Findings, Teacher Observational Findings, Teacher 
Survey Findings , and Principal Interview Findings. Under each subheading, I discuss relevant data from that data 
source. 

Student Findings 

A total of 817 sixth-grade students responded to the baseline student survey in October, while 795 sixth- 
grade students responded to the final student survey in May. The survey consisted of 59 items across 6 scales. Table 
1 details these scales. 



Table 1. Distribution of Teacher Survey Items across Scales (with Reliabilities) 



Focus of Scale 


Number of Items 


Reliability 1 


Student technology skills 


18 


.871* 


Student comparison to "typical" students on technology skills 


3 


Student feelings about school and learning 


10 


.625 


Student opinions about technology 


9 


.630 


Student ratings of how technology makes them feel 


11 


.772 


Student preferences for classroom technology uses 


8 


.778 


TOTAL ITEMS: 


59 





* NOTE: For purposes of reliability calculations, the first pair of scales was combined. 



Table 2 summarizes student responses to items in the six scales used in the survey. Student responses to 
each scale are discussed beneath the table. 



Table 2. Mean Response Data for Baseline and Final Student Survey (by Scale) 



Focus of Scale (highest possible rating) 


Baseline 

Mean 

Rating 


Final 

Mean 

Rating 


Net 

Change 


% 

Change 


Student technology skills (5) 


2.43 


2.61 


+.18 


+3.5% 


Student comparison to "typical" students on technology skills (3) 


2.32 


2.38 


+.06 


+1.9% 


Student feelings about school and learning (4) 


3.14 


3.14 


0 


0% 


Student opinions about technology (4) 


3.22 


3.26 


+.04 


+1.1% 


Student ratings of how technology makes them feel (5) 


3.98 


3.95 


-.03 


-0.4% 


Student preferences for classroom technology uses (5) 


3.76 


3.77 


+.01 


+0.3% 



Note: Highest possible rating for each scale indicated in parentheses after scale focus description. Higher ratings are better. 



As Table 2 illustrates, responses to the first scale went up dramatically, while two other scales went up 
moderately, and three stayed largely the same. Although Table 2 shows percentage of change for each scale baseline 
to final, rather than cite results solely by percentages, from this point on differences will be discussed in terms of 
effect size (ES). The effect size describes how great an effect a significant change between measurements suggests 
has taken place. That is, it goes further than a simple significance rating to identify effects in terms of their 
magnitude. An effect size of .2 is considered small , while an effect size of .5 is considered medium , and effect sizes 
of .8 or greater are considered large. 

The increase in ratings on the first scale across 1 8 separate technology skills is statistically significant (ES 
= +.32). This scale employs five points, with one representing technology skills students report they do not have 
and five representing self-reported skills at a level where the student usually helps other students complete a task 
involving technology. On the baseline, students reported they often needed help to perform almost all of the 
technology tasks listed. Only four items were rated above 3.0 (often do alone), and these included using word 
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processing (3.37), turning the computer on and off (3.29), searching the Web (3.28), and using a printer (3.27). 

Thus, sixth graders apparently did not feel they had much skill in completing most technology tasks. 

In contrast, on the final survey seven items were above 3.0 and eight items had increases that were 
significantly higher (Effect sizes ranging from +.27 to +. 46 ). The final survey results suggest students now viewed 
themselves as able to handle many technology tasks independently ( often do alone or help others). The technology 
tasks they see themselves needing help on or not doing include using databases (2.49), making graphics and editing 
pictures (2.42), using CD-ROMS (2.24), making videos (1.83), taking digital photos (1.73), editing videos (1.70), 
doing email (1.60), and using a scanner (1.51). A number of these were technology tasks not currently completed in 
these schools and thus may have received lower values because many respondents answered don y t do. 

Ironically, when asked to compare their skills to those of "typical” students, the surveyed sixth graders 
described themselves on the baseline as about one-third of the way between about the same and better (mean = 2.32 
out of 3.0). Thus, although they did not rate themselves as strong in completing these technology tasks, they 
appeared to believe that other students were not much stronger. Despite the significant increases in students’ 
assessments of their independence in completing technology tasks by the end of the study, students did not raise 
their assessment of comparison to “typical” students much (mean = 2.38 out of 3.0). 

Students were asked to rate how they feel about school and learning. On a four-point scale, they rated 
themselves on the baseline at a mean of 3 . 14 , the same mean rating as on the final survey. This means that they rated 
most statements in this section of the instrument as being usually true of them. The statements in this section with 
which students agreed most strongly included School prepares me well for the future (means = 3 . 53 , baseline; 3 . 59 , 
final) and When I do well in school, it is because I have tried hard (means = 3 . 53 , baseline; 3 . 51 , final). The two 
items with which they most disagreed were When I do badly in school, it is the teacher’s fault ( 2.30 baseline, 2.34 
final) and How I do in school is completely out of my control ( 2.43 baseline and 2.41 final). 

The next section of the instrument asked students their opinions about technology. On a four-point scale 
(not true! to always true!) the mean for items in this section was 3.22 on the baseline and 3.26 on the final survey. 
This means student respondents on average rated the statements on this scale as about one-quarter of the way 
between usually true and always true! Two items in this scale had effect sizes were equal to or greater than .2: 
Students assigned a mean rating of 3.42 to I can do assignments when I have to use the computer , up .17 from the 
baseline (ES = +. 20 ) and The teachers in my school know how to use technology went down . 1 5 from the baseline to 
3.13 on the final survey (ES = -. 20 ). One of the highest rated items in this section on both baseline and final survey 
was/wfl«/ to use a computer more (means = 3.60 and 3 . 57 , respectively). The items on this scale with which they 
disagreed most on both baseline and final survey were Computers are a waste of time (means = 2.17 and 2 . 11 ) and 
Technology makes it harder for me to get my work done (means = 2.35 and 2 . 27 ). 

Students were asked next to respond to statements about how technology makes them feel. On a five-point 
scale from never (1) to all of the time (5), students assigned a mean rating of 3.98 to statements in this section on the 
baseline and 3.95 on the final survey. This suggests they felt the statements were true most of the time. The highest 
rated items on both baseline and final was I like using computers in class (means = 4.51 and 4.48 respectively). The 
items with which students expressed the greatest disagreement were When we use computers, I feel left out (means = 
1.17 baseline and 1.15 final), I get nervous when we will be using technology (means = 1.28 and 1 . 21 ), and I have 
trouble with the software we use in class (means = 1.65 and 1 . 58 ). Only one item on this scale had a baseline-to- 
final effect size of .2 or greater: I’m excited when I get to use a new computer program. This item went from 3.92 
(baseline) to 3.66 (final), producing an effect size of -. 20 , suggesting a faded novelty effect. 

The final section of the student baseline survey asks students to tell how much they like particular uses of 
technology in the classroom. They were asked to rate such uses on a five-point scale from I don’t like it! (1) to It’s 
great! (5). Across the eight items in this section, students assigned a mean baseline preference of 3 . 76 , about three- 
quarters of the way from It’s OK to It’s good. This mean rating went up slightly to 3.77 on the final survey. The uses 
they rated most highly included going online to look for facts or materials ( 4.06 and 4 . 11 , respectively) and doing 
research on the computer to complete an assignment ( 4.01 and 4 . 08 , respectively). Only one item on the scale 
showed a baseline-to-fmal change that met the .2 effect size threshold: My group uses a computer to prepare a 
report or presentation. The mean for that item went from 3.79 on the baseline to 4.08 on the final, making it one of 
the best liked activities (ES = +. 25 ). It is interesting to note that these top three uses are ones facilitated by having 
individual laptops. 

Teacher Observational Findings 

As noted earlier, all sixth grade teachers were observed by the same classroom observer, using a 
standardized observation protocol. The focus of the observation was the extent to which technology was 
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incorporated into the observed lesson. Table 3 presents the proportion of time in which computers were used in the 
classroom during the observation. Table 4 shows observer ratings of teacher apparent comfort on the final 
observation in using technology, while Tables 5 and 6 show ratings for instructional effectiveness and holistic 
assessments on final observations. 

Table 3. Observed Computer Use by 42 6 th Grade Teachers (in minutes, divided by observation period/activity) 





School 1 


School 2 


School 3 


School 4 


TOTAL | 


Activity 


Baseline 

<n=1) 


Final 

(n=5) 


Baseline 

<n=3) 


Final 

(n=4) 


Baseline 

(n=0) 


Final 

(n=6) 


Baseline 

(n=1) 




Baseline 

(n=5) 


Final 

(n=23) 


Whole-class Use 


1.5 


31.5 


87.5 


11.0 


0 


85.0 


27.5 


0 


116.5 


127.5 


Student Team Use 


0 


44.0 


0 


62.5 


0 


0 


0 


73.0 


0 


179.5 


Individual Student Use 


0 


19.0 


0 


26.0 


0 


73.5 


0 


138.5 


0 


257.0 


Students Learning Tech 


0 


0 


20.0 


0 


0 


0 


0 


0 


20.0 


0 


Teacher Administrative Use 


0 


6.5 


0 


0 


0 


0 


0 


18.0 


0 


24.5 


Handling Tech Problem/Setup 


0 


5.0 


2.5 


3.5 


0 


0 


0 


19.0 


2.5 


27.5 


No Computer Use 


48.5 


144.0 


40.0 


97.0 


0 


141.5 


22.5 


151.5 


111.0 


534.0 


Total 


50.0 


250.0 


150.0 


200.0 


0 


300.0 


50.0 


400.0 


250.0 


1150.0 



Table 4. Final Observation Mean Ratings of Teacher Apparent Comfort with Technology (divided by task) 



Observation Item Task 


Applicable # of 
Teachers Observed 


Mean Rating 
(of a possible 4) 


Setting up equipment. 


9 


3.00 


Handling minor technical problems. 


12 


3.00 


Maintaining lesson flow while changing from one technology to another. 


14 


3.57 


Helping students use technology. 


13 


3.38 


Using traditional technologies (overhead, slide projector, tape recorder) 


6 


3.67 



Table 5. Final Observation Mean Ratings of Apparent Instructional Effectiveness of Computer Uses 



Observed Computer Use 


Applicable # of 
Teachers Observed 


Mean Rating 
(of a possible 5) 


Word processing 


9 


3.89 


Using teaching/learning programs 


9 


4.00 


Searching the Internet/Web 


9 


4.44 



Table 6. Final Observation Mean Holistic Assessments of Instructional Use of Technology 



Observation Item Task 


Applicable # of 
Teachers Observed 


Mean Assessment 
(of a possible 5) 


Students appeared comfortable using technology. 


20 


4.85 


Technology used seemed well matched to instructional tasks to be accomplished. 


20 


4.80 


There was enough technology available to accomplish intended tasks. 


21 


4.76 


When working with technology, students seemed self-directed. 


20 


4.70 


Students appeared engaged by the lesson. 


21 


4.62 


Instructional pacing felt smooth with technology causing few interruptions. 


20 


4.60 


Technology not used for its own sake, but to enhance lesson. 


21 


4.48 


Use of technology did not cause discipline or classroom management problems. 


21 


4.43 


Teacher appeared well focused on lesson and little distracted by technology. 


20 


4.25 


Students appeared excited about using technology. 


20 


3.75 



As Table 3 shows, while only one school had more than one teacher observed using technology at baseline, 
all four schools had multiple technology-using teachers at final. In fact, there were 4.6 times as many teachers 
observed using technology at final than at baseline and these teachers were observed using technology for 4.4 times 
as long. Perhaps equally important, the pattern of technology use observed was very different at final, with more 
individual student and team use as opposed to the whole-class used observed on the baseline. Tables 4 through 6 
demonstrate that final observations revealed teachers who were more comfortable using technology and who used 
technology in ways well matched to instructional goals without disrupting pacing or creating classroom management 
problems. The observer holistically concluded that students were engaged by these technology-using lessons and 
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technology enriched the lessons (as opposed to simply being an add-on for technology’s sake). The observer was 
asked to assess equity of access and use and concluded opportunity was about the same for student regardless of 
gender or race. The observer did conclude, however, that in terms of perceived academic level, students at lower 
perceived levels of academic ability actually had slightly more opportunity to use technology than other students. 
Teacher Survey Findings 

The teacher survey was made up of 10 scales, ranging in reliability from +.746 to +.903 (Cronbach’s 
alpha). Table 7 shows the change baseline to final for these 10 scales. 

Table 7. Baseline and Final Teacher Survey Mean Response Data by Respondent Groups 

(Aggregated by Scale, So rted by All Respondents Effect Size) 

All Respondents 6 th Grade 7 th Grade Special Education 

(n=103) (n=42) (n=43) (n=18) 





Baseline 


Final 


Effect 


Baseline 


Final 


Effect 


Baseline 


Final 


Effect 


Baseline 


Final 


Effect 


Scale 


Mean 


Mean 


Size 


Mean 


Mean 


Size 


Mean 


Mean 


Size 


Mean 


Mean 


Size 


Student Independence Using 
Tech / Comparison to 
“Typical” Students 


57.93 


68.79 


0.83 


59.61 


70.00 


0.84 


60.76 


70.21 


0.81 


47.67 


65.08 


1.28 


Teacher Role in Student 
Success / How Much 
Difference Teacher Makes 


92.39 


85.58 


-0.74 


92.27 


84.64 


-0.91 


93.17 


86.65 


-0.72 


90.78 


86.86 


-0.36 


Previous Training’s Value in 
Preparing for Tech Use/ 
Value of Types of Future 
Training 


70.16 


63.77 


-0.53 


69.31 


67.23 


-0.20 


70.77 


60.93 


-0.76 


70.50 


61.25 


-0.76 


Quality of Tech Support 


10.10 


11.06 


0.38 


10.02 


11.18 


0.42 


10.17 


11.00 


0.35 


10.11 


11.07 


0.42 


Integration of Computers in 
the Classroom 


37.32 


40.33 


0.33 


39.00 


40.88 


0.23 


37.26 


38.53 


0.13 


33.83 


42.71 


1.00 


General School Support for 
Technology Use 


32.63 


34.36 


0.31 


33.27 


35.67 


0.47 


33.14 


34.36 


0.20 


29.82 


32.71 


0.68 


Student Use of Types of 
Software 


35.70 


37.50 


0.20 


37.35 


38.25 


0.10 


34.79 


34.65 


-0.02 


33.77 


42.07 


0.87 


Beliefs about Technology’s 
Value in Education 


81.04 


79.71 


-0.16 


82.29 


80.13 


-0.26 


80.76 


81.13 


0.05 


79.00 


76.38 


-0.37 


Teacher Confidence and 
Comfort with Technology 


49.81 


50.18 


0.05 


49.90 


50.69 


0.12 


51.29 


51.32 


0.00 


45.81 


47.57 


0.25 


Teacher Use of Types of 
Software 


38.31 


38.38 


0.01 


39.62 


38.13 


-0.17 


37.92 


37.47 


-0.05 


35.69 


39.67 


0.47 



As the left column in that table illustrates, 7 of 10 scales had significant changes baseline-to-fmal. Highest 
among these was enhanced student technology competence and independence, as well as their comparison to typical 
students. Teachers were asked to compare the technology skills of their students to “typical” students. Teachers 
rated students on a three-point scale {worse, about the same, better). As Table 8 shows, teacher ratings came up 
significantly. 



Table 8. Baseline and Final Mean Teacher Response for Comparison to r ‘Typical ” Scale (with Effect Size) 



Item 


Baseline 
Mean Rating 


Final 

Mean Rating 


Net Change 


Effect Size 


In using computers, my students are... 


2.21 


2.40 


+.19 


.31 


In using software, my students are... 


2.09 


2.27 


+.18 


.32 


In learning new technology, my students are... 


2.18 


2.35 


+.17 


.29 



Two scales went down significantly, Teacher ’s role in student success / How much difference the teacher 
makes and Previous training's value in preparing for technology use / Value of future training. All but one of the 15 
items on the Teacher role / Difference scale went down. This seems to suggest that teachers perceived themselves as 
being less effective or making less of a difference as the year went on. Some teachers call this the “wear-down” 
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factor: As the year goes on, they become less optimistic and feel less effective. Over the summer many teachers then 
recharge and come back optimistic again in fall. The fact that this scale was significantly lower at final should not 
surprise us. It suggests instead that technology does not enhance the teachers’ sense of impact. As further evidence 
that this scale may be measuring a technology-independent variable, it is worth noting that -in the depths of snowy 
and cold February — teachers actually rated this scale lower than at final. The Previous training/ Future training 
scale lists seven possible sources of previous preparation to use technology and asks the respondent to rate the extent 
to which he or she acquired skills from that source. The scale next lists 17 different types of training and asks the 
respondent to rate the potential benefit of each on a five-point scale. Of the 7 sources listed, only one source showed 
a significant increase in teachers attributing their computer skills to its role. That source was in-service courses and 
workshops and showed a mean change from 2.85 on the baseline to 3.23 on the final (ES = +. 51 , medium). Of the 17 
different types of training listed, all showed marked drops in perceived benefit baseline-to-fmal. The mean 
differences ranged from -.22 to -. 57 . There were still five types of training that are at or above the middle position 
(; moderate benefit) and that these five are ones that were among the top rated types on the baseline as well. While all 
17 types rated at (or within .02 of) at least small benefit on the baseline, two types of training -introductory 
computer basics and word processing basics — were now below that level, suggesting a sizable portion of teachers 
rated them as of no benefit. In general, all “basic” training now seemed of lower perceived benefit. 

Teachers rated quality of tech support significantly higher on final than on baseline and, as Table 9 shows, 
two of the scale’s three items had notable effect sizes. 



Table 9. Baseline and Final Mean Teacher Response for Quality of Tech Support Scale (with Effect Size) 



Item 


Baseline 
Mean Rating 


Final 

Mean Rating 


Net Change 


Effect Size I 


1 have enough access to tech support to resolve hardware, 
software, and tech use problems. 


3.01 


3.48 


+.47 


.46 


1 [do not] have to contact our tech-support person several times 
before 1 get assistance.* 


3.29 


3.62 


+.33 


.31 


The on-site tech-support person is committed to getting the 
technology running and keeping it running well. 


3.81 


3.96 


+.16 


.17 



* Note: Words in brackets indicate item was reversed on the survey. 



The Classroom Computer Integration scale lists a variety of ways to use technology and asks the teacher to 
rate frequency of use on a five-point scale from not at all (1) to every day (5). Table 10 shows the baseline and final 
means, net change, and effect sizes for all 12 items. It lists items in descending order, based on final reported 
frequency of use. 



Table 10. Baseline and Final Mean Teacher Ratings for Classroom Computer Integration Scale (with Effect Size) 



Item (Uses) 


Baseline 

Mean 


Final 

Mean 


Net 

Change 


Effect 

Size 


Independent learning (students working alone on individual computers) 


3.93 


4.05 


+.12 


0.12 


As a research and class preparation tool for you 


3.84 


4.01 


+.17 


0.15 


Individual instruction (one student on one computer) 


3.67 


3.96 


+.29 


0.27 


As an instructional resource for students (Websites, online files) 


3.38 


3.65 


+.28 


0.26 


As a research tool for students 


3.11 


3.45 


+.35 


0.33 


As a productivity tool for students (create charts, reports, other products/artifacts) 


3.28 


3.37 


+.09 


0.09 


As a communication tool for you and your students (email, electronic discussion) 


2.97 


3.33 


+.35 


0.21 


As a problem-solving tool for students 


3.11 


3.28 


+.17 


0.16 


Cooperative groups (teams working on computers) 


3.05 


2.92 


-.13 


-0.11 


As a classroom presentation tool for you and/or your students 


2.56 


2.79 


+.22 


0.19 


Whole class instruction (one computer as central focus) 


2.37 


2.69 


+.31 


0.25 


Small group instruction (teaching subgroup of students using a single computer) 


2.08 


2.61 


+.54 


0.43 



Only one of 12 uses has lower reported frequency on the final than on the baseline: use by cooperative 
groups , and that effect size is less than .2 (as well as the reported final frequency still hovering close to the once a 
week point on the scale). In fact, it appears to be a verified common use of computers by sixth-grade teachers, since 
use of computers by a team of students made up 15.6% of observed computer use during the final set of observations 
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(see Table 3). The two most frequent reported uses are for independent learning by students and as a teacher 
research and class preparation tool. On the five-point scale, these equate to several times a week. These were the 
two most frequent uses reported on the baseline as well. Although both final frequency means are higher than on the 
baseline, the change does not meet the .2 threshold. The next use is individual instruction and teachers reported it 
was also used close to several times a week. The increase from baseline to final was significant, with an effect size 
of +.27. This seems to confirm what the observer found in final observations, where individual student use of 
computers constituted 22.3% of observed computer use. The next five uses were reportedly used more than once a 
week, but less than several times a week. Of these five, four had effect sizes larger than +.2. Of the final four uses in 
the list that were reportedly used less frequently than once a week, two had effect sizes greater than +.2. These two, 
whole class instruction and small group instruction , are the least frequently reported uses of computers, but both 
increased notably from baseline. In particular, small group instruction showed the greatest change, moving from 
once a month or less to almost two-thirds of the way to once a week. 

Going back to Table 7, we find the next largest scale effect size is for general school support for 
technology use (ES = +.31). This ten-item scale addresses time, access, nature of training, and administrative 
encouragement. Respondents were asked to rate their agreement with a series of statements, using a five-point scale 
from strongly disagree (1) to strongly agree (5). Six of the 10 items met the .2 effect size threshold, all positive (see 
Table 11). While these six statements showed significant baseline-to-fmal increases, it is perhaps useful to examine 
the final means as a way of gauging teacher end-of-year sentiment. Respondents moved substantially closer to agree 
in terms of their rating of administrative support for training. This suggests an enhanced perception of such support. 
The next three statements, however, were ones where respondents are only one-third or so higher than neutral in 
their final mean responses. This suggests either that many respondents were undecided on these statements or 
perhaps that some group of teachers was in greater agreement, while another was in lesser. The final two statements 
in Table 1 1 are ones that have moved from halfway between disagree and neutral to two-thirds of the way. While 
this is desirable, once again, it suggests at least some sizeable portion of respondents still disagreed with these 
statements. Thus, it appears that there may have been room for improvement in teacher perception of general school 
support. 



Table 11. Baseline and Final Mean Teacher Ratings for Key Items from General School Support Scale 
(with Effect Size, Ordered by Final Mean) 



Item 


Baseline 

Mean 


Final 

Mean 


Net 

Change 


Effect 

Size 


The administration supports technology-related training. 


3.70 


3.87 


+.17 


.22 


My school gives me enough tech-related support. 


3.09 


3.39 


+.31 


.32 


1 have had adequate training in using computers. 


3.12 


3.36 


+.24 


.22 


Training is offered often enough to meet my needs. 


2.97 


3.29 


+.32 


.34 


Teachers [do] have enough time to develop good instruction using technology. * 


2.46 


2.71 


+.26 


.26 


1 have enough time to learn computer skills 


2.45 


2.68 




.23 



* Note: Words in brackets indicate item was reversed on the survey. 



On the student use of types of software scale teachers were asked to tell the frequency with which their 
students used 15 types of software. The scale offered five points, from not at all (1) to every day (5). Eight of these 
items showed significant changes in reported frequency baseline-to-fmal. Table 12 lists all 15 uses, ordered by final 
survey reported frequency of use. The top four student uses remain the same baseline-to-fmal, although all show a 
marked and significant increase in reported use. Reported student use of drill-and-practice software jumped ahead of 
educational games (whose reported use was unchanged). The single greatest change was in reported use of 
presentation software, which is now used approximately once a month. Graphics software exhibited a similar rise in 
frequency of use and is also now reportedly used about once a month. The three types of software reported as having 
the lowest frequency of use were simulations, movie-making, and collaborative writing. These same types of 
software were rated as three of the least frequently used pieces of software on the baseline (and on the midpoint). 
While simulations showed lower reported student use on the final than on the baseline, that difference had an effect 
size less than .2. Only one use showed lower reported frequency of use at that level on the final survey: 
collaborative writing software. Based on teacher comments, it appears this software was little used during the study. 
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Table 12. Baseline and Final Mean Teacher Ratings for Student Software Use 
(with Effect Size, Ordered by Reported Frequency) 



Item (Type of Software) 


Baseline 

Mean 


Final 

Mean 


Net 

Change 


Effect 

Size 


Word processor (AppleWorks, Open Office, etc.) 


3.81 


4.02 


+.21 


.23 


Web browser (Internet Explorer, Netscape, etc.) 


3.67 


3.91 


+.25 


.26 


Online search engines (Google, Yahoo, MSN, etc.) 


3.35 


3.70 


+.34 


.31 


Websites with downloadable images or files. 


2.96 


3.24 


+.28 


.22 


Drill and practice (quiz programs, self-test and practice programs, multiplication drills, etc.) 


2.58 


2.89 


+.31 


.25 


Educational games (Tom Snyder products, History Concentration, Alien Rescue, etc.) 


2.89 


2.89 


+.01 


0 


Presentation software (Keynote, PowerPoint, etc.) 


1.88 


2.36 


+.48 


.58 


Graphics (iPhoto, Photoshop, GraphicsConverter, etc.) 


1.91 


2.22 


+.30 


.31 


Teaching and learning programs (Geometer’s Sketchpad, MasterKey, etc.) 


2.00 


2.09 


+.09 


.08 


Spreadsheet (AppleWorks, Open Office, etc.) 


1.93 


2.07 


+.15 


.15 


Database (AppleWorks, Open Office, etc.) 


1.73 


1.78 


+.05 


.05 


Audio capture / editing (Audio Recorder, Finale Notepad, QuickTime, etc.) 


1.54 


1.67 


+.13 


.15 


Simulations (Sim City, Sim Ant, etc.) 


1.66 


1.54 


-.12 


-.12 


Movie-making (iMovie) 


1.50 


1.52 


+.02 


.03 


Collaborative writing (SubEthaEdit) 


1.74 


1.52 


-.21 


-.21 



Principal Interview Findings 

Although principal interviews used a structured form and this same form was used with both principals and 
assistant principals, the data gathered tend to be more divergent. That is, because of the interactive interview 
process, respondents sometimes headed in directions that differed slightly from those initially intended by a 
question. At the same time, such divergence tends to produce a richer data set. While some questions were common 
to all three interview forms, each interview also used different questions focused on identifying changes occurring as 
the year progressed. This section summarizes the three interviews holistically, emphasizing changes as appropriate. 

When asked in October, principals said one would see little technology use in classrooms in the building, 
but this changed progressively to the point at final that they felt technology was widely used and widely integrated 
into the classroom ethos. This matches what student and teacher surveys and classroom observations suggested. As 
might be expected, principals noted technological issues as serious problems as the beginning but these became less 
a concern as the year went on. Many principals noted teacher anxiety as a “technology-related problem” in the early 
stages, but not at final. This suggests the teachers were indeed becoming more comfortable working with technology 
and more confident in their abilities and the abilities of their students. 

Principals noted that the laptop initiative appeared to be leading to better teacher planning for classes and a 
change from teacher-centered, more authoritative models of teaching to more student-centered, distributed- 
knowledge models. Principals noted that the availability of online resources and content seemed to have enriched the 
instructional environment considerably, reducing reliance on the textbook. Better time management was also cited, 
since the use of online resources like First in Math™ and other similar activities made it possible to use even small 
time segments for instruction, as well as offering the chance to keep all members of the class engaged at their own 
instructional level while addressing the needs of subgroups of students in the classroom. Principals suggested this 
led to more hands-on time and more time-on-task than was previously possible. Principals also noted that students 
were beginning to move toward seeing a need for communication: They read and write to persuade, with technology 
helping them present their arguments. This seems to confirm the dramatic increase in use of presentation software 
revealed in both survey and observation. 

In disagreement with teacher surveys, principals felt there was insufficient support for technology use. 

They appeared, however, less focused on nuts-and-bolts tech support and more focused on support for curricular 
integration of technology and enhanced teaching/learning practice. They saw the beginnings of use of technology to 
differentiate instruction and were anxious to see this use enhanced. One principal noted that the greatest benefit of 
the initiative was staff development: Teachers were being exposed to new instructional techniques and technology 
was being used to facilitate the implementation of these techniques. All principals wanted more such in-service 
training, feeling that it did much to enhance the effectiveness of the teaching staff, regardless of technology use. 
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Principals were asked to put a dollar amount on the value to each individual student of having a laptop 
computer for the entire year. This question was a challenge for them. Three of the five classified its value differently 
according to the user or use, suggesting a range of value. The responses of the five respondents who were able to 
assign a dollar figure ranged between $250 per student and $3333 per student (mean range: $1300 to $1500 per 
student). They felt the value depended on the nature of the student and the use, suggesting students who lacked 
access outside school or who were more challenged academically perhaps benefited more than other students. As 
one principal said, “It’s the equalizer; it puts every kid on the same level.” They noted that the initiative had 
equalized access and instructional practice across the schools, eliminating what some perceived as pre-existing 
inequalities among the four middle schools. 

Principals suggested the laptop initiative produced more engaged teachers and learners and more 
enthusiasm among students for learning. Principals suggested this was not based on the technology itself, but rather 
on the access to resources it provided and the changes in teaching practice that had accompanied the use of 
technology. Principals also noted that students seemed better equipped with real-world skills than they had been 
before the initiative was implemented. Principals universally praised the school board for its courage in supporting 
the laptop initiative, suggesting the initiative should be continued. 

Discussion and Interpretation 

The final survey data support the conclusion that tech support currently meets teachers’ perceived needs. 
Although principals cited a variety of minor glitches and logistic problems in all three sets of interviews, the 
teachers almost universally seemed pleased with both the level of support they received and the responsiveness of 
those who supply it. Given the size of this initiative and the demands it places on a school district, this is certainly 
something to which the district may point with pride. That teachers and principals now wanted to move further and 
get deeper into curricular applications and integration is a favorable sign. 

Special education teachers appear to have embraced this initiative. They showed significantly more 
favorable reactions across a wide range of items on both midpoint and final survey and rated their students as 
working more independently on the midpoint than the baseline and on the final than on the midpoint. Analysis of 
baseline-to-fmal changes show the greatest changes among all teachers surveyed, changes that suggest special 
education teachers were using technology, their students were using technology, and both special education teachers 
and students were growing in confidence and independence. There is much discussion in the literature about how 
technology can help special populations, in many cases such help is limited by the amount of technology available to 
them. If special education teachers were correct in their assessment that students were able to perform more tasks 
independently without teacher help, this offers the possibility of providing a kind of ‘intellectual robotics’ that 
enables special education teachers to do more with technology than they would be able to do without it. 

Anecdotal reports from the principals indicated students may be more engaged by laptop-enriched learning 
experiences. Several principals also suggested each student having a laptop may enable teachers to do some basic 
differentiated instruction, mostly through pacing and individualized materials. Principal comments implied support 
for curricular integration continued to be the most pressing need in the school district. They indicated the greatest 
potential of the technology may be in differentiating instruction to address the needs of a much wider range of 
students than most teachers were currently able to address. In fact, in final interviews, some principals even 
suggested the district might consider moving from individual laptops to class sets, freeing previously committed 
funds for use in enhancing professional development for curricular integration and differentiated instruction. 

In any school district, at least some portion of the population will likely leave school prior to graduation 
from high school. Student ratings of their own technology abilities went up significantly on the posttest measure and 
students viewed themselves as even more capable than “typical” students. If teacher and student assessments of 
student independence using technology are accurate, this initiative appeared to be producing at least some of what 
are termed 21st Century Technology Skills in middle school students. This should enhance their real-world 
employment opportunities, regardless of how much farther they go in the system. 

Clearly, an initiative like this one is expensive. Most principals suggested having a laptop was of greater 
value to some students, for some subjects, and when used by certain kinds of teachers. Should the district have to 
reduce the scope of the laptop initiative in future, this may offer a starting point for hard decisions. For instance, 
exactly which students benefit the most and why? Which subjects lend themselves most to academic enhancement 
through use of laptops? And what are the characteristics of teachers who make the most effective use of laptops with 
their students? 

When principals were specifically asked to judge whether the laptop initiative was a success or not, they 
unanimously rated it as a success, although they noted ways in which its implementation might be improved in 
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future. As one of the principals noted in the midpoint interviews, it is not technology that makes the difference; it is 
solid pedagogy. Thus, while having laptops may enhance instructional opportunity, it cannot, in and of itself, 
enhance academic achievement. The principals suggested, however, that discontinuing the initiative would have 
negative effects on instructional practice and morale, as well as act to reinstate some of the educational inequities 
that apparently existed before the initiative was implemented. Several principals also suggested at least some 
segment of the parent population would be disturbed, perhaps because of lost opportunities for their children, 
perhaps because a school laptop represents almost the only way their children can gain access to such technology. 
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Background Information 

Interaction is considered to be a necessary and important ingredient for a successful learning experience so 
distance learning practitioners and researchers have concerned with how much interaction a distance learning 
environment could provide for students (Mclsaac & Gunawardena, 1996). Vygotsky (1978, 1986) asserts that a 
great deal of learning occurs in a social context and is brought out by interactions with other people. The process of 
interactivity among learners brings in the benefits including analyzing, synthesizing, and evaluating course content 
(Lave, 1991), increases in learning achievement (Bates, 1993), applying higher levels of cognitive processing 
(Garrison, 1993), and development of collaborative and cooperative learning skills (Berge, 1995). Computers can 
potentially contribute to a sense of community within the group and create a social bond which offers important 
motivational and cognitive benefits in learning (Harasim, 1992). 

Western culture values individualism; pedagogical practices are designed for developing individualism and 
individuated skills while Chinese culture is highly collectivist and pedagogical practices tend to reflect the 
importance of the group (Carson, 1992). As a result of culture differences, online interaction practices in distance 
learning within Western culture may not be beneficial to Chinese students. 

The purpose of this study was to find out Chinese students’ attitude toward their interactions with peers, 
instructors, course material, and interface in a cooperative online distance education environment. The kinds of 
interaction facilitating Chinese students’ learning, the barriers encountered, and suggestions to overcome these 
barriers were investigated in the study. 

Activity theory was applied in analyzing Chinese students’ interactions in a cooperative distance learning 
environment. Activity theory (Leont’ev, 1978; Vygotsky, 1978) is increasingly being used to explain social aspects 
of technology- supported learning (Jonassen, 2002). Activity theory can contribute to computer-supported 
collaborative learning by “[understanding] learning not as the internalization of discrete information or skills by 
individuals, but rather as expanding involvement over time — social as well as intellectual — with other people and 
the tools available in their culture” (Russell, 2002, p. 65). 

Activity theory has its origins in the social-historical approaches around 70 years ago and can be 
characterized in (a) objective, (b) ecological, and (c) socio-cultural perspectives on human activity (Kaptelinin, 

1996). The key elements of activity theory related to computer- supported cooperative learning are shown in Figure 1: 

Instruments 




Figure 1. Engestrom ’s classic model of Activity Theory. The model shows the relationship between the subject, the 
object and the community, as well as how rules, instruments, and the division of labor are used in transforming the 
object into the desired outcome (Engestroem, 1987). 
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There are seven elements in the model and the definition of each element as follows (Jonassen, 2002; Jonassen 
& Rohrer-Murphy, 1999; Collis & Margaryan, 2004): 

1. Subject: The individuals such as leaner, peers, facilitator, supervisor, instructor, and mentor who 
participated in the activity. 

2. Instruments: Methods, resources, supports, online tools and environments that facilitate the activity such as 
technological tools, networks, and learning resources. 

3. Object: Products created by the subjects during the activity such as learning tasks, assignments, and 
projects. 

4. Community: Socio-cultural environment in which interactivity takes place such as virtual classroom and 
organization. 

5. Rules: Standards and norms of the community that rule the activity such as frameworks, culture and other 
standards that influence the learning environment. 

6. Division of Labor: Roles and relationships within the community that affect task division and 
responsibilities. For example, roles and relationships within cooperative teams and courses. 

7. Outcome: The overall results achieved by the activity system such as final products and learning outcome. 

From the prospective of Activity theory, the process of cognition is no longer studied based on certain aspects 
individually but on the interaction between learners, tools, resources, and context that influence learning in a socio- 
cultural setting. 



Methodology 

A purposeful sampling was used to select participants in University of Central Florida who: (1) have taken 
at least 2 online or mixed mode courses; (2) have used the function of discussion board or chat room; and (3) must 
be over 18 years old. Four in-depth interviews were conducted to give an in-depth portrait of their participation. In 
order to completely understanding the feelings of the participants, the qualitative research was conducted for 
researcher to go deep into the interviews. The whole interview was recorded. The audio-tape was then transcribed by 
the researcher and a Grounded Theory Analysis based on Activity theory was done on the transcripts. 

Data was gathered in the fall 2006 semester and consisted of: (1) Four face-to-face audio-taped semi- 
structured interviews lasted for 30 to 50 minutes; and (2) Four field notes, 5 analytic memos and more than 20 
journal entries. Recordings were transcribed and compared with field notes, analytic memos and researcher’s journal 
for details. The data was analyzed using Grounded Theory methods. 

Results and discussion 
The results that organized according to Activity Theory indicate: 

Subject — Subject 

A number of participants said that course mail was commonly used in Chinese students’ cooperative online 
interaction and it was helpful to their learning. They felt that they were comfortable to interact with others by this 
asynchronous communication tool because they were allowed to have more time understanding what others said and 
have sufficient time to respond to others. The participants expressed that: 

“I like email, I usually ask questions and if we had teamwork, I discussed with my teammates 
and asked professor questions using emails.” 

“we usually used email to contact with others. We used email to exchange word documents, 
draft, something like that. ... I like email because I can have more time to think about what I 
want to say and I have more time to check their feedback and response.” 
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Asynchronous online discussion boards were also commonly used and they were regarded to be a helpful tool in 
Chinese students’ cooperative online interaction. A participant said that: 

“. . .the discussion board is useful for me to post our reflection for the class and our projects. 

For me and for that class, the discussion board most of the time is to post something but 
sometimes we can get feedback from classmates.” 

Rules — Instruments 

However, synchronous online discussion tools such as chat room were not regarded to be useful to Chinese 
students according to their cooperative learning experience. All participants expressed the similar experience that 
they didn’t like to use chat room due to flaws in technology and language barrier. 

“I don’t like chat room is that I think the design of the program is not good at all. I mean if 
there are more than three or four [people participated], if more people talk at the same time, 
the screen will refresh very quickly and it’s hard to track what was said.” 

“I can not type English very fast so sometimes I felt that I can’t catch up [with] their speed. 

Sometimes when I finish my typing, they already jump to the next topic or next issue. So let 
me feel like kind of, you know. . .embarrassed or upset.” 

Subj ect — Community 

As for Chinese students’ attitude toward cooperative online interaction, in general, most of them had 
passive attitudes toward cooperative online discussions. They basically didn’t actively take part in discussions unless 
they were required to do so. However, compared to face-to-face discussions, they felt more comfortable and were 
more willing to express themselves in online environments. 

“I don’t feel stressed in posting discussions. The professors set up due dates and I posted 
when required. If I don’t post it I won’t get grade.” 

“For me, if you don’t ask me to [express my ideas] I won’t do it but I think for somebody, if 
you ask them to do, I think Asian students will express themselves more freely if they are 
required to in the online environment than in face-to-face class.” 

Division of Labor — Instrument 

To improve cooperative online interaction and make Chinese students’ learning experience a successful one, 
most of them agreed that instructors’ skills in designing activities and facilitating interactions were essential abilities 
in promoting successful cooperative online distance education interaction. 

“I believe that [a comfortable online discussion environment] depends on how the professor 
designed the courses. The way the professor designed that could make me comfortable to 
express my ideas. I think it’s the feeling to make you feel comfortable and you will be more 
willing to talk.” 

“I think the professor’s ability to use chat room was important. If the professor could keep 
an eye on all students in the chat room and facilitate some students. I think that’ll be 
better.” 

Subj ect — Instrument 

To facilitate Chinese students’ cooperative online distance education interaction, participants recommended 
integrating more technology features such as video and audio into cooperative online learning environment to help 
them correctly interpret others’ ideas and efficiently communicate with others. 
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“I think maybe they can use some kind of video chat so I can see others’ faces to feel more 
reality not just see words.” 

“I would like to say to integrate video and audio communication tools there, I think that’ll 
be much better and much efficient and effective to communicate.” 

Conclusion 

According to Chinese students’ cooperative online learning experience, interacting with other peers and 
instructor was frequently taken place and asynchronous communication tools such as email, course mail, and 
discussion boards contributed to their interactivity and learning. However, due to barriers such as language and 
cultural difference, Chinese students were not fully comfortable in interacting with others using synchronous 
communication tools. They needed more time to organize their ideas, as well as understand others’ and needed time 
to compose their feedback. The interface of these tools and instructors’ facilitation skills were considered to be an 
important role in successful cooperative online learning interaction. Because of culture difference, Chinese students 
would not actively participate in cooperative online learning interactions unless they were asked to do so. Besides 
instructor’s facilitation skills, integrating technological communication features such as video and audio into chat 
room was recommended to improve cooperative online learning interaction. To better serve Chinese students’ 
learning needs and help improve their learning outcomes, instructional designers and instructors should be aware of 
Chinese students’ experience of interaction in cooperative online distance education. 
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Abstract 

The purpose of this study was to examine the efficiency of three different interface types on Web-based 
instruction: a text-based interface, a graphical interface and a metaphorical interface. In order to determine which 
interface type reduces cognitive load, learning efficiency scores formulated with performance scores and mental 
effort levels from 50 undergraduate students were compared. Results indicated there was no difference among the 
groups in terms of learning efficiency. Based on the results, implications for instructional designers are presented. 

Introduction 

In Web-based instruction, the user interface is a communication point between an instructional unit and a 
learner. While instructional designers are making efforts to develop effective and aesthetically pleasing interfaces, 
designing a usable and appealing interface, in fact, is still challenging for instructional designers (Metros & Hedberg, 
2002; Chalmers, 2003). The importance of the user interface in Web-based instruction has been emphasized by 
many researchers: attracting learners’ attention (Hron, 1998; Szabo & Kanuka, 1998), facilitating communication 
between a learning domain and learners (Metros & Hedberg, 2002; Parizotto-Ribeiro & Hammond, 2005), and 
reducing cognitive load (Chalmers, 2003; Haag & Snetsigner, 1993; Hannafm & Hooper, 1989). 

Cognitive load has been measured by a learner’s difficulty level or time to completion. In addition, a 
learning efficiency metric has been used to quantify the efficiency of instruction (e.g., Clark, Nguyen & Sweller, 
2005; Tuovine & Paas, 2004). The efficiency score is calculated by performance and mental effort, meaning a high 
efficiency condition occurs when performance is higher and effort is less (Tuovinen & Paas, 2004). Based on 
cognitive load theory, researchers and practitioners have proposed interface design guidelines (e.g., Norman, 1998; 
Shneiderman & Plaisant, 2005; Swan, 2001). The following guidelines have been proposed to reduce cognitive load; 
(a) provide informative system feedback (Norman, 1998; Shneiderman & Plaisant, 2005); (b) use intuitive elements 
(Norman, 1998); (c) provide directions (Swan, 2004); (d) avoid extraneous objects (Beriswill, 1998; Swan, 2004); 

(e) use organizational strategies (Chalmers, 2003; Norman, 1998); (f) provide visual elements (Swan, 2004); and (g) 
enhance learner’s autonomy (Shneiderman & Plaisant, 2005; Swan, 2004). 

A cognitive load perspective is used to evaluate how well the user interface supports the cognitive 
processes involved in the task (Plass, 1998), and the goal of the approach is the interface should impose as minimal 
as possible a cognitive load on the learner (Stoney & Wild, 1998). In a graphical user interface, graphical elements 
have been used to make Web pages pleasing in order to get users’ attention. In addition, the elements use metaphors 
to assist users to intuitively navigate through a system (Lang, 2003). While menu buttons or images employ different 
metaphors to reflect each function in a graphical interface, metaphorical interface design has been proposed to 
connect whole-screen or system-environment to the instructional content (Haag & Snetsigner, 1993). A metaphorical 
interface design focuses on creating an interactive environment that reflects the learning content in order to provide 
learners with instructional cues. Metaphor plays a significant role in scaffolding the learner to intuitively interact 
with multimedia resources (Hron, 1998; Lang, 2003), because it helps the learner structure links between theme and 
content (Cates, 1996; Hron, 1998). Like graphical user interfaces, metaphorical interfaces can also decrease 
cognitive load and disorientation (Hron, 1998). However, there is little empirical evidence for the benefits of 
graphical or metaphorical interface regarding learning outcomes and cognitive load. Therefore, the purpose of this 
study was to examine if the interface of Web-based instruction affects learners’ performance and cognitive load with 
three different interface types: a text-based interface, a graphical interface and a metaphorical interface. 
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Methodology 



Three different types of user interface were implemented with the same content on Cognitive Information 
Processing theory. The process of developing these three instructional units consisted of two layers: content layer 
and interface layer (Beriswill, 1998). First, in the content layer, learning materials, which were presented in the form 
of various media types, such as images and interactive sample experiments, were developed to provide the learners 
with cohesive learning. The same learning materials, including informational graphics and interactions, were used in 
all three instructional units. Next, in the interface layer, design guidelines based on various studies were applied to 
implement three different user interfaces as shown in Figures 1, 2 and 3. These instructional units provided different 
navigation designs, but the basic mechanism was same in that the units had both linear and user-controlled 
navigation. 



(fe Cognitive Information Processing | Shorterm Memory - Mozilla Firefox 
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Short-term Memory 



eg Rehearsal (1/1) 

What happens to information while it is in working memory determines whether--and how-it will get stored in 
long-term memory. We can hold things in working memory for a while by "rehearsing" them. An example of this 
is when we repeat a phone number or person's name to ourselves just long enough to dial a number or make an 
introduction. After we've used the information, it will probably be lost to us. 

Rote rehearsal or maintenance rehearsal can help you to keep information in short term memory for more than 
30 seconds. BUT, if anything happens to interrupt your rote rehearsal, the information will be lost unless you 
have already succeeded in moving the information into long term memory. You can probably think of real life 
examples where this has happened: you were trying to keep a telephone number in your head, but someone 
interrupted your thoughts, and you lost the number forever. 



Back Next 



Figure 1. Text-based interface implemented with typography and basic design guideline (Lee & Boling, 1999) 




Figure 2. Graphical interface using five guidelines: unity, proportion, homogeneity, balance, rhythm (Parizotto- 
Ribeiro, Hammond, Mansano & Cziulik, 2004) 
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Figure 3. Metaphorical interface with primary and secondary metaphor (e.g., Cates, 1994; Lang, 2003; Hron, 1998) 

Various data were collected from fifty undergraduate students: a pretest score, time to complete the 
instruction, a difficulty level of the instruction, a posttest score and responses for the attitude survey adapted from 
the studies of Ciavarelli (2003) and Moreno and Valdez (2005). The difficulty level, which can be called mental 
effort rating, was rated by the question adapted from Kalyuga and Sweller (2005): “Please indicate how difficult the 
test you just took was by clicking on the appropriate degree of difficulty.” The participants had to select one of the 
nine degrees ranging from extremely easy to extremely difficult. Cognitive load was measured with the time to 
completion, the mental effort for instruction, and the responses to the questions in the attitude survey. The 
mental effort and score difference between pretest and posttest were used to quantify learning efficiency. 

Results 



First, pretest scores and post test scores were analyzed with one-between, one-within mixed analysis of 
variance (ANOVA). There was no difference of learning performance regarding interface types while all instructions 
were effective in increasing test scores (F=2.356, p=.109). 

Second, three variables were used to compare cognitive load in the three interface groups: (a) mental effort 
for the instruction, (b) time to completion for the instruction and (c) attitude survey data. The mental effort scores 
were obtained when students rated the degree of difficulty on a scale of 1 to 9. The analysis of variable (ANOVA) 
revealed that there was no difference among groups in terms of mental efforts and time to completion (Time to 
completion: F=.123, p=.884; mental efforts: F=.609, p=.549). 

Third, the performance score and mental effort for the instmction were used to quantify learning efficiency. 
This analysis followed Paas, Tuovinen, van Merrienboer and Darabi (2005) process for measuring efficiency. 
Regarding the performance score, differences between the pretest and posttest scores were used in the learning 
efficiency formula because the pretest scores among groups were different. The two data were standardized since 
they were scored with different scale, and learning efficiency score was computed by the formula as follows: 



= p-m e 



(£ = efficiency, P = performance, Mg= mental effort) 



The graphical interface had the highest efficiency score (£=.1240) as compared to the text-based interface (£=- 
.0882) and the metaphorical interface (£=-.0052, but the analysis of variance (ANOVA) revealed that there was no 
difference among groups in terms of efficiency scores (F =101, p=.904). 

Finally, the answers to the six questions in the attitude survey regarding cognitive load showed that there 
was no difference other than the question number 4 asking if the interface captured learners’ attention (F=8.400, 
p=.001). The mean scores and standard deviations presented in Table 1. 



52 



Table 1 



The mean scores of the attitude survey questions 



Questions 


Text-based 


Graphical 


Metaphorical 


1. It took too long to complete. 


3.62(1.12)* 


3.38 (1.39)* 


3.63 (0.60)* 


2. There is too much information on the pages. 


3.69(1.03)* 


3.00(1.41)* 


3.42 (0.90)* 


3. Graphics or other elements on the pages are distracting. 


4.54 (0.52)* 


4.38(0.51)* 


4.26 (0.45)* 


4. Elements on the page, such as heading and graphics, 
focused my attention. 


2.77(1.01) 


3.15(0.90) 


4.00 (0.75) 


5. The information layout and locations are consistent 
throughout the instruction. 


4.08 (0.95) 


3.69 (0.75) 


4.05 (0.78) 


6. The menu is consistent throughout the instruction. 


4.15(0.99) 


3.77 (0.84) 


4.16(0.38) 



a Standard deviations are given in parentheses 

* The three values of the question 1 , 2 and 3 were recoded so that higher number represents positive perception. 



Discussion and conclusion 

This study was designed to explore the effects of interface on Web-based instruction. It should be noted that 
all interface types applied design guidelines based on the previous research and convention. The hypothesis was that 
a metaphorical interface would be the most effective and efficient interface, but the results revealed that there was 
no significant difference among groups in terms of performance or cognitive load. The implications of this research 
are informative to instructional designers as they develop Web-based instruction. First, this study revealed that 
interface did not exert any influence on learning performance, so instructional designers and developers should 
focus on creating meaningful instruction with interactions because the learners depend on more these elements than 
on the interface. Second, interface did not affect learners’ cognitive load and learning efficiency. An aesthetically 
pleasing interface did not make any difference in cognitive load. It implies that time-consuming works of making 
attractive interface may not be worth the investment. 

Even though the results indicated that both graphical interface and metaphorical interface were not 
beneficial to learning, the study requires further investigation with different assessment, domain and learning 
materials. For example, the assessment in this study was focused more on declarative knowledge, so a different 
assessment which includes more procedural or application knowledge might produce different results. A concrete 
domain instead of the abstract domain that was used in this study could be examined with different interface types. 

In addition, one type of interface with different types of learning materials could investigate to determine which 
combination of interface type and learning material is more effective. 
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Abstract 

The primary goal of the study is to analyze learner behaviors during collaborative group work in a digital learning 
community. A qualitative data analysis program, NVivo 2, was used to code the vast amount of online transcripts 
and written interviews based on the grounded theory. Resulting codes were aligned in the four subsystems of 
activity theory and the activation of subsystems by group work phases was investigated. Conflicting factors and 
facilitating factors while college students achieve a common learning goal were identified. At the same time, 
whether the high performing groups show patterns of learner behaviors, conflicting factors, and facilitating factors 
that differ from those of the low performing groups was examined. 

Introduction 

A digital learning community (DLC) is an emerging instructional approach that embraces the characteristics 
of collaborative learning and computer-mediated communication in networked environments. A DLC draws 
attention because it provides students with opportunities to extend their learning experiences by sharing their new 
ideas with, and receiving critical and constructive feedback from, community members (Palloff & Pratt, 2005). 

Also, learning together in a DLC provides chances for students to improve collaboration and communication skills 
that are required on the job (Bennett, 2005). Furthermore, teamwork is another generic skill developed in higher 
education (Candy, Crebert, & O’Leary, 1994). In recognizing the benefits of a DLC, the questions that come to mind 
are “How do learners in a DLC collaborate to achieve a common learning goal?” and “How do we need to design 
and support such nontraditional pedagogies of learning?” 

Despite the promising benefits of collaborative learning, learners experience tensions from mixed feelings of 
wanting to learn independently and a fear of being isolated from the community. Dirkx and Smith (2005) argued that 
these negative experiences are derived from “ambivalence.” A major focus of Computer- Supported Collaborative 
Learning (CSCL) has been providing better ways of understanding learners in communication and collaboration to 
achieve learning goals. Group synergy created by collaboration, however, is not fully explained by CSCL theory and 
still remains as abstraction (Stahl, 2006). 

The direction of research has been geared toward two aspects of CSCL, that is, outcomes and processes. 
Research examined the effectiveness of different tools, techniques, and learner outcomes in collaborative learning. 
On the other hand, the process-oriented research examined socio-cultural factors and learners’ language acts 
(Treleaven, 2004). Activity theory has been used to understand human behaviors in a social context and is one of the 
major theories on which CSCL is based. 

This study attempted to shed light on the process of online collaboration with activity theory. The 
investigation revealed different patterns of learner behaviors during collaborative group work which is aligned with 
the framework of activity theory. In particular, both facilitating and conflicting factors were identified after 
analyzing the data from online transcripts and semi- structured interviews. This study also sought to determine if any 
differences exist between high and low performing groups for both factors in their collaboration activities. 



Theoretical Background 

Activity theory is a philosophical and multi-disciplinary framework to research various forms of human 
behaviors. It has been used as a socio-cultural analytical framework in social contexts with humans and mediators 
(Kuutti, 1996; Jonassen & Rohrer-Murphy, 1999). The applications of activity theory are found in learning (Hung & 
Wong, 2000; Barab, Barnett, Yamagata-Lynch, Squire, & Keating, 2002; Johri, 2005; Barab, Schatz, & Scheckler, 
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2004), human-computer interaction (Kuutti, 1991; Nardi, 1995), instructional design (Jonassen & Rhorer-Murphy, 
1999) and work practices (Engestrom & Middleton, 1996). 

The root of activity theory stems from three historical origins: classical German philosophy from Kant to 
Hegel, the writings of Marx and Engels, and the Soviet Russian cultural-historical psychology of Vygotsky, 
Leont’ev, and Luria (Engestrom, 1987). Activity theory has evolved and reached the third generation. The first 
generation of activity theory stems from the idea of mediation by Vygotsky. The second generation of activity 
theory was derived from Leont’ev’s work. He made distinctions between an automatic operation, an individual 
action, and a collective activity. The third generation of activity theory has expanded to include the activity system 
by Engestrom (1987). 

Activity theory has been further developed as a practical model of human activity, an activity system. An 
activity system contains six interacting components: subjects, objects, tools, rules, division of labor, and community. 
Activity systems are organized to achieve the goals of activities of the activity subsystems (production, exchange, 
distribution, and consumption subsystems) that describe functions, interactions, and relationships between the six 
components. The production subsystem explains how subjects transform the object of the activity system into the 
outcome. The exchange subsystem shows how subjects are constrained by rules and interact with the community in 
accordance with the rules. The distribution subsystem describes how the community defines a division of labor for 
the subject to accomplish the object of the activity system. Lastly, the consumption subsystem shows how the 
subject and the community around the subject collaborate, and also how the community consumes effort from the 
subject (Engestrom, 1987; Jonassen, 2000). 



Research Questions 

The study intended to answer the following questions: 

1 . What are the different patterns of learner behaviors in a digital learning community? 

2. What are the emerging conflicting factors in a digital learning community? 

3. What are the emerging facilitating factors in a digital learning community? 

4. How do the high performing groups differ from the low performing groups in learner behaviors, conflicting 
factors, and facilitating factors? 



Method 



Participants and Setting 

In order to examine collaborators’ behaviors in an online environment, we chose six groups who enrolled in a 
college-level class, titled ‘Information Society and Education,’ in the fall semester of 2006 at a large university in 
Seoul, Korea. Though the class met offline every week, each group of four members also worked independently 
online on the group project. Ranking each group according to performance, we selected three groups to form the 
upper half and the other three to form the lower half. Each group selected an instructional design method, submitted 
a project plan, and implemented an online course. At the end of the semester, each group presented the website they 
implemented to the class. This research used online transcripts of 24 students and semi- structured interviews of 
seven participants. 

Procedures 

The constant comparison method was used to capture real phenomena. As a result, the codes were created from 
the raw data. The qualitative data analysis program, NVivo 2, was used to code consistently the online transcripts 
from six groups for fifteen weeks. The coding scheme was divided into three main categories. The first category was 
learner behaviors that represented the specific behaviors performed by participants. The second category was 
conflicting factors and the third was facilitating factors. The codes in the learner behaviors category was then further 
categorized into the related subsystems (production, distribution, exchange, consumption subsystems) defined in 
activity theory. Upon completion of the coding scheme, the codes and the frequencies of the codes found in both the 
upper and lower halves were compared. 
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Results 



First, the analysis identified 29 different types of learner behaviors. These 29 open codes generated by the 
grounded theory revealed seven themes of learner behaviors in a digital learning community, as shown in Figure 1. 
The seven themes were information seeking (i.e., share material), extraction of relevant information (i.e., summarize 
material), idea generation (i.e., suggest an idea, request an idea, collect ideas, ask questions), co-construction (i.e., 
outline tasks, suggest a meeting, suggest group work, request to do work, etc.), division of tasks (i.e., divide tasks, 
redistribute tasks), making or conforming to rules (i.e., suggest a rule, share template, remind of schedule, raise an 
issue, etc.), and evaluation (i.e., evaluate material, evaluate self or peer work). The two most frequently observed 
categories were information seeking and co-construction. 




Figure 1. Emerging themes of learner behaviors in a DLC 



Those 29 different learner behaviors were aligned with the four subsystems of the activity system based on 
activity theory. Among the four subsystems (production, consumption, distribution, and exchange subsystems) in the 
activity system, the consumption subsystem had eleven different behaviors (i.e., share material, suggest an idea, ask 
questions, etc.) and the highest frequencies of observations. The exchange subsystem had eight different behaviors 
(i.e., suggest a rule, remind of schedule, evaluate peer work, etc.) and the production subsystem showed five 
different behaviors (i.e., modifying material, submitting reports, writing meeting minutes, etc.). The production 
subsystem had more incidents than the exchange subsystem even if it had fewer types of behaviors. Lastly, the 
distribution subsystem showed one type of behavior, dividing tasks, and showed the least number of incidents. 

Figure 2 shows how active each subsystem is according to the project phase. 
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Figure 2. Activated subsystems by group work phases 

Second, six different categories of conflicting factors emerged: inefficiency of work, unfamiliarity, difficulty in 
communication, issues of roles, conflicting schedules, and technical difficulties (Table 2). The factor most 
frequently appeared was inefficiency of work. The reasons of inefficiency included lack of skills, lack of group 
rules, applying inefficient methods, and lack of necessary resources. The next most frequently mentioned factor was 
difficulties of communication in online environment. This factor included uncertainty, nonparticipation, difficulty 
with relationships, and delayed feedback. Other conflicting factors included role-related issues, unfamiliarity, 
schedule conflicts, technical difficulties, etc. When the observed conflicting factors were matched with the 
components in the activity system, the most frequently observed conflicting factors resided between subjects and 
tools components. 

Table 2 

Emerged conflicting factors 



Categories 


Open codes 


CODE 


Definitions 


Inefficiency of 
work 


Lack of skills 


IWLS 


Work is not performed efficiently 
due to a team member’s lack of skills 
regarding tools (e.g., Photoshop, 
Flash) required to complete the group 
project 




Applying inefficient 
methods 


IWAI 


Work is not performed efficiently 
due to the fact that a team member 
used inefficient methods to complete 
the group project 




Lack of resources 


IWLR 


Work is not performed efficiently 
due to lack of resources to complete 
the group project 




Lack of group rules 


IWLG 


Work is not performed efficiently 
due to the absence of rules defined 
by team members to complete the 
group project 




Difficulty with finding 
the relevant info 


IWDF 


Work is not performed efficiently 
due to the fact that team members do 
not know how to find relevant 
information 


Unfamiliarity 


Unfamiliarity with 
processes or methods 


UF_UP 


Frustrations due to unfamiliarity with 
how to proceed with the group 
project and with what methods to use 




Unfamiliarity with 
topics or material 


UFTP 


Frustrations due to unfamiliarity 
regarding the project topic or the 
relevant material 
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Difficulty in 
Communi- 
cation 


Uncertainty 


DC UN 


Difficulty in communication caused 
by not understanding what other 
team members meant exactly 




Nonparticipation 


DCNO 


Difficulty in communication due to 
the fact that a team member did not 
participate in a decision making 
process 




Difficulty with 
relationships 


DCDR 


Difficulty in communication due to 
discomfort among other team 
members especially when there are 
age differences 




Delayed feedback 


DCDF 


Difficulty in communication caused 
by the nature of asynchronicity of 
online communication 


Issues of 
roles 


Work delays 


IRWD 


Issues of roles due to the fact that a 
team member does not complete 
one's assigned work on time 




Varying levels of 
contribution 


IRVC 


Issues of roles due to the fact that a 
team member recognizes inequality 
of efforts made by each team 
member 




Issues of role 
assignment 


IRRA 


Issues of roles due to the fact that 
roles were not assigned equally. 


Conflicting 

schedules 


Conflicts with other 
personal commitments 


CS_PC 


Conflicting schedules among team 
members due to jobs, part-time work, 
or other personal commitments. 




Conflicts with other 
subjects/exams 


CSEX 


Conflicting schedules among team 
members due to other subjects or 
exams 


Technical 

difficulties 


System issues 


TDSY 


Issues with sharing files due to the 
learning management system 




Corrupted or 
incompatible files 


TDFL 


Issues with sharing files due to 
corrupted files, or incompatibilities 
between different versions of 
software, etc. 



Third, the analysis revealed five facilitating factors: efficiency of work, effective communication, the 
competence of team members, group cohesiveness, and goal orientation (Table 3). The most frequently observed 
factor was group cohesiveness. This category was composed of intimacy, a sense of community, and encouraging 
others. When the observed facilitating factors were placed in relevant components in the activity system, the most 
frequently observed facilitating factors were located between subjects and community components. 

Lastly, comparing the high performing groups with the low performing groups, no difference was found in 
terms of types of learner behaviors. One evident difference was that the high performing groups revealed about 40% 
more of such incidents. When the learner behavior codes were aligned in the activity system, the consumption 
subsystem was the most highly activated subsystem, followed by the production, exchange, and distribution 
subsystems, in that order. A salient difference between the high and low performing groups was that the frequency 
of the behaviors in the consumption subsystem was in decline at the project completion phase for the high 
performing groups, whereas that of the low performing groups was increasing. As for facilitating and conflicting 
factors, the high performing groups revealed more incidents of conflicting factors and fewer incidents of facilitating 
factors. 
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Table 3 

Emerged facilitating factors 



Categories 


Open codes 


CODE 


Definitions 


Efficiency of 
work 


Applying 
efficient methods 


EWAE 


Work is performed efficiently because 
team members use efficient methods to 
do the group project 




Conforming to 
rules 


EWCR 


Work is performed efficiently because 
team members follow the rules defined 
by the team to do the group project 




Efficient role 
assignment 


EWFR 


Work is performed efficiently due to 
efficient role assignment based on team 
members’ strengths 


Effective 

communication 


Timely decision 
making 


ECDM 


Team members make a decision 
through responsive communication 




Honesty 


ECHO 


Team members talk straight regarding 
the way or the quality of peer work 




Proactiveness 


ECPR 


Team members show eagerness or take 
an initiative in communication. 


Competency of 
team members 


Responsibility 


CTRE 


Team members are responsible for the 
assigned task or the project overall 




Competency of 
tools 


CTCT 


Team members are competent in using 
tools required to complete the group 
project 




Previous 

experience 


CTPE 


Team members have previous 
experience or prior knowledge to do the 
group project 


Group 


Intimacy 


GC IN 


Team members feel close to each other 


cohesiveness 


Sense of 
community 


GC SC 


Team members have a sense of 
community, referring to the group as 
’we’, ’us’, or ’our’ 




Encouraging 

others 


GCEO 


Team members encourage each other to 
keep up the good work 


Goal oriented 


Sense of 
competition 


GOSC 


Team members feel a sense of 
competition with other teams 
(performance goal) 




Excellent 

outcomes 


GOEX 


Team members strive to create 
excellent outcomes (mastery goal) 



Discussion and Implications 

This study produced a coding scheme that can be used to analyze learners’ collaborative behaviors. To generate 
a coding scheme we have used a mixed approach that is grounded in empirical data and theory based (activity 
theory). The results of this study showed how production, exchange, distribution, and consumption subsystems were 
activated during collaborative work. Each subsystem can be considered as a learning space in a digital learning 
community. This study revealed that the consumption subsystem was the most highly activated subsystem 
throughout the project phases. DLC design should be able to support learner behaviors identified in the consumption 
subsystem. Likewise, when learners engage in the production subsystem, necessary tools or artifacts should be 
available in the DLC environment. In addition, rules govern the learner behaviors during group work. DLC 
designers should take into account rules that facilitate effective and productive learning. Lastly, division of labor 
should be designed to ensure both individual accountability and optimal interdependency among team members. 

Based on the conflicting factors found in this study, some implications can be made. First, there is a need for 
different implementation strategies depending on the nature of conflicting factors. It was evident that some of the 
conflicting factors are not as harmful but can serve as an alert for interventions. The fact that the high performing 
groups revealed higher incidents of conflicting factors supports Engestrom’s point (2002) that development occurs 
when contradictions are overcome. Apparently, other types of conflicting factors are harmful and can lead to 
negative learning experiences or deficient learning outcomes. These include work delays, varying levels of 
contribution, issues of role assignments, and nonparticipation. An instructor or a facilitator should proactively 
monitor and intervene by using them as indicators. 
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Recommendations for Future Research 



Based on the results of this study, the following future research is suggested. First, the findings of the study 
have drawn a few implications in providing an effective DLC learning environment. Those DLC design implications 
are required to be validated and refined through design-based research. Second, the study implied two different 
kinds of conflicting factors. This interpretation requires in-depth research regarding specific conflicting factors and 
how these factors can affect student achievement or learning processes in a DLC. Third, group cohesiveness was the 
most frequently observed category of facilitating factors. In this study, group cohesiveness is not directly related to 
group performance. Further research on group cohesiveness and performance would be beneficial. Lastly, the digital 
learning community in this study was formed rather involuntarily due to the nature of higher education. Future 
research could replicate this study in an autonomously formed learning community to determine if it reveals 
different patterns of learner behaviors, conflicting factors, and facilitating factors during group work. 

References 

Barab, S.A., Barnett, M., Yamagata-Lynch, L., Squire, K., & Keating, T. (2002). Using activity theory to understand 
the systemic tensions characterizing a technology-rich introductory astronomy course. Mind, Culture, and 
Activity, 9, 76-108. 

Barab, S.A., Schatz, S. & Scheckler, R. (2004). Using activity theory to conceptualize online community and using 
online community to conceptualize activity theory. Mind, Culture, and Activity, 11(1), 25-47. 

Bennett, S. (2005). Supporting collaborative project teams using computer-based technologies. In T. Roberts (Ed.), 
Online collaborative learning: Theory and practice (pp. 1-27). Hershey, PA: Information Science. 

Candy, P., Crebert, G., & O’Leary, J. (1994). Developing lifelong learners through undergraduate education. 

Canberra, Australia: Australian Government Publishing Service. 

Dirkx, J., & Smith R. (2005). Learning to learn in online collaborative groups. In T. Roberts (Ed.), Online 
collaborative learning: Theory and practice (pp. 132-159). Hershey, PA: Information Science. 

Engestrom, Y. (1987). Learning by expanding: An activity theoretical approach to developmental research. Helsinki, 
Finland: Orienta-Konsultit Oy. 

Engestrom, Y. & Middleton, D. (Eds.) (1996). Cognition and communication at work. Cambridge, UK: Cambridge 
University Press. 

Engestrom, Y. (2002). The activity system. Center for activity theory and developmental work research. University 
of Helsinki. Finland. 

Hung, D. & Wong, A. (2000). Activity theory as a framework for project work in learning environments. 

Educational Technology, 40(2), 33-37. 

Johri, A. (2005). Online, offline and in-between: Analyzing medicated-action among american and russian students 
in a global online class. In T. Roberts (Ed.), Computer- supported collaborative learning in higher education 
(pp. 283-312). Hershey, London, Melbourne, Singapore: Idea Group Publishing. 

Jonassen, D. H. & Rohrer-Murphy, L. (1999). Activity theory as a framework for designing constructivist learning 
environments. Educational Technology Research and Development, 47(1), 61-79. 

Kuutti, K. (1996). Activity theory as a potential framework for human computer interaction research. In B. Nardi 
(Ed.): Context and consciousness: Activity theory and human computer interaction (pp. 17-44). Cambridge: 
MIT Press. 

Kuutti, K. (1991). Activity theory and its applications to information systems research and development. In H.E. 
Nissens, H.K. Klein, & R. Hirschheim (Eds.), Information systems research: Contemporary approaches and 
emergent traditions (pp. 529-549). North Holland: Elvsevier Science Publisher B.V. 

Nardi, B.A. (1995). Studying context: A comparison of activity theory, situated action models, and distributed 
cognition. In B.A. Nardi (Ed.), Context and consciousness: Activity theory and human-computer interaction 
(pp. 69-102). Cambridge, MA: MIT Press. 

Palloff, R. M., & Pratt, K. (2005). Collaborating online: Learning together in community. San Francisco, CA: A 
Wiley Imprint. 

Stahl, G. (2006). Group cognition: computer support for building collaborative knowledge. Cambridge, 
Massachusetts: The MIT Press. 

Treleaven, L. (2004). A new taxonomy for evaluation studies of online collaborative learning. In T. Roberts (Ed.), 
Online collaborative learning: Theory and practice, (pp. 160-180). Hershey, PA: Information Science. 



61 




Instructional Strategies to Influence Self-Efficacy for Technology Integration: 
A Study of Middle-School Preservice Teachers 

Liesl M. Combs 
Michael A. Evans 
Virginia Tech 



Introduction 

Middle schools are increasingly confronted with a broad mix of new and legacy digital technologies and 
personal media devices - handheld computers in classrooms, cell phones in break rooms, and portable gaming 
devices on playgrounds. The anticipated ubiquity of technological resources in schools, as evidenced by more and 
more multifunctional, multipurpose wireless and mobile devices (take the iPhone for example) coming to market, is 
driving teacher educators to re-examine preservice preparedness in light of such trends. In response, teacher 
preparation educators have come to realize a significant need to shift purpose toward providing properly trained 
teachers who are prepared for the influx of new digital technologies and media. A noticeable conundrum is that tech 
savvy students outside the classroom become befuddled when asked to use same technologies for instructional 
purposes. Thus, a noted issue in this area is the inconsistent level of technology integration among pre- and in- 
service teachers. As explained in the literature, the inconsistency can be partially attributed to trends in acceptance 
of technology within schools along with attitudes toward and competency in technology for classroom use (Brzycki 
& Dudt, 2005; Finley & Hartman, 2004; Pope, Hare & Howard, 2005). With pre-service teachers reporting low 
perceived competency levels, anxiety elevates as the demand to integrate through classroom practice becomes 
imminent. Anxiety therefore, remains a barrier to technology acceptance, most significantly in the early phases of 
technology adoption, when pre-service teachers receive training (Christiansen & Knezek, 2002). 

To more thoroughly understand reasons behind this issue - low levels of perceived competency followed 
by high levels of anxiety, and thus poor integration performance, we focus on the need to research and develop 
instructional strategies for middle school teacher preparation programs. The perceptions of preservice teachers 
toward technology have been widely identified as a target area; instructional strategies to influence these perceptions 
being seen as a crucial area for further investigation (Delcourt & Kinzie, 1993; Glazewski & Brush, 2005; 

Hakkinen, 1995; Reed & Overbaugh, 1993). Research, in regards to teacher preparation, is primarily designed to 
make improvements and determine best practices for meeting standardized requirements. In response to the 
increasing importance of state and national standards, a general concern surrounds understanding how teacher 
preparation is changing to yield more teachers prepared for such responsibilities (Combs & Evans, 2007). As Topper 
(2004) indicates, “The current movement towards standards for technology in teacher education provides an 
opportunity to begin to study how teachers’ changing technology competence also influences their plans for 
integrating technology into their classroom practices” (p. 303). Researchers have studied the effects of instructional 
strategies and program design to determine some of the most effective strategies being used to incorporate the 
fundamental technology component (Brzycki & Dudt, 2005; Collier, Weinburgh & Rivera, 2004). Combs & Evans 
(2006) argue that potential benefits from instructional strategies designed to increase self-efficacy include improved 
performance, increased confidence, excitement about future opportunities, and a heightened desire to uphold and 
increase knowledge gained through training. The pressing pedagogical question is when and how to instruct 
preservice teachers on effective, creative uses of instructional technologies and digital resources. Previous research 
has indicated that increased self-efficacy positively effects future technology integration into classroom practice. 

Self-efficacy beliefs provide a strong source of motivation for and predictor of technology use (Hill, Smith, 
& Mann, 1987; Jorde-Bloom, 1988; Mclnemey, Mclnemey, & Sinclair, 1994; Milbrath & Kinzie, 2000). The 
attitude toward a given task can severely affect the amount of effort exerted and therefore affects the desirability of 
the outcome of the tasks and actions: learners must believe learning will influence and enhance their future and 
opportunities (Bandura, 1977; Driscoll, 2005, p. 317). This results in a change in self-perception and therefore 
influences a learners’ desire to proceed in the learning process, practice, and apply skills. Instructional strategies 
developed for pre-service teachers to affect a learners’ perceived self-efficacy and the effect on the transfer has been 
identified as a significant line of research to pursue (Ertmer, 2005). Consequently, our research question is this: 

What effects do instructional strategies designed specifically to influence self-efficacy have on preservice teacher 
perceptions of technology integration in the classroom? 
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Method 



For this study, we employed a participatory action research method (Kemmis & McTaggart, 2000). The 
participants (n=14) were members of a cohort (n=22) of middle school pre-service teachers at a small university in 
southwest Virginia. The cohort members were involved in the final year of a teacher preparation program and 
actively participated in their school placements. Members of the cohort were also enrolled in an instructional 
technology methods course that was required for degree completion. Fourteen of the twenty-two students enrolled in 
the course gave written consent to participate in the research as directed through IRB procedures. Of the fourteen 
participants, 1 1 were female and three were male with a mix of graduate and undergraduate as traditional and non- 
traditional students. 

In consideration of resources available within local school districts and needs expressed by practicing 
teachers, the first author (as instructor-of-record) designed the instructional technology course with several critical 
components in mind. First, it was important to display the realities of local classrooms (i.e. avoid sharing technology 
not available or entirely beyond school budget). Second, it was essential to set realistic goals and begin with simpler 
technologies. Third, participants needed just-in-time instruction, meaning it was important to help them develop 
skills that could be immediately applied to their current classroom experiences. In taking all of these points into 
consideration, the overall instructional strategy of the course was problem-based in nature; the instructor provided 
tools and equipment where participants were encouraged to investigate applications of the equipment into their 
current instructional planning. As an instructor, the focus was on providing opportunities for participants to become 
familiar with and share ideas on using equipment; the focus for the participants was to explore their content areas 
and grade levels and design instructional activities around their planning needs. The activities throughout the course 
were purposed to encourage participant discovery with the intent of increasing self-efficacy toward technology use 
for instruction. In addition, the activities involved using equipment, exploring web applications and general 
programs with the intent of increasing the probability of transfer to their instructional planning and classrooms. 

For the purpose of analysis and discussion, we focus solely on one activity designed with a mental models 
approach to learning (Magliaro & Shambaugh, 2006). As Combs and Evans (2006; 2007) point out, when 
individuals lack relevant prior knowledge, they are less likely to assimilate or accommodate new knowledge. Mental 
models, which can be described as executable mental simulations (Mason-Mason & Tessmer, 2000), are significant 
in the formation of new knowledge that can lead to success and thus increased self-efficacy. In this activity, both 
participants and the instructor had a significant role in the execution of learning activities. The description of this 
activity, the role of participants, and the role of the instructor are described as follows: 

Activity Description 

For a class period of approximately two hours, the participants rotated through five stations, each having 
several pieces of instructional technology. The stations, as illustrated in Table 1, were completely disassembled and 
all cords and accessories were packed in cases. Participants were instructed to determine the purpose of the station 
and required to work to connect the equipment until they had an operable product (see Figure 1). 
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Figure 1 : Participants beginning activity at Station 3 



As an example, station one had an LCD projector and a visual data presenter. At this station, the 
participants needed to use all the equipment and cords in order to present items displayed on the visual presenter to a 
larger group. In order to complete the activity, participants were required to visit all five stations (see Table 1) with 
at least minimal success (for this activity, mastery was not a requirement). 



Table 1 : Activity stations and their purposes 





Station 1 


Station 2 


Station 3 


Station 4 


Station 5 


Equipment list 


LCD projector 


LCD projector 


LCD projector 


LCD projector 


LCD projector 




Visual data 


Kodak EasyShare 


Dell laptop 


DVD/VCR 


Dell laptop 




presenter 


digital camera 


Intel QX5 


combo 


Interwrite 








Digital Blue 


Speakers 


SchoolPad 








microscope 






Purposes 


- To display 


- To display 


- To display 


- To display 


- To display 




items placed on 


photos/videos 


images/videos 


videos to large 


contents of a 




the visual 


taken with the 


captured 


groups 


computer to 




presenter to large 


camera to large 


through the 


- To explore 


large groups 




groups 


groups 


microscope to 


alternative 


- Optional: to 




- To learn basic 


- To explore 


large groups 


methods of 


learn basic 




operation of the 


alternative forms 


- To learn basic 


video display in 


operation of the 




visual presenter, 


of display in the 


operation of the 


the absence of a 


SchoolPad 




explore usage 


absence of a 


digital 


television or 






potential 
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Role of the Participant 

Participants were directed to collaborate with two or three other participants, thus forming groups of three 
or four, with which to rotate through each of the five stations. As part of the activity, one member from each group 
was responsible for writing complete instructions on how their group connected the equipment successfully (one set 
of instructions per station yielding three or four separate sets of instructions from each group). For their instructions, 
participants were encouraged to consider the perspective of a novice user and use language and vocabulary 
appropriate for such an audience. An example given was a new teacher who may have been in their second or third 
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career, thus being part of an older generation of learners not as readily exposed to the equipment found in the 
activity. Therefore, the language and vocabulary used in the written instructions should provide a sense of clarity 
and simplicity to avoid confusion and frustration for the user. As the groups rotated, they were video and audio 
recorded at stations one and three. 

Role of the Instructor 

For this specific activity within the course, the instructor was careful to take participant experiences and 
observed typical experiences of many teachers into consideration. For example, it was important to present 
equipment that is most readily available and/or most commonly found within school districts. The purpose here was 
to provide participants with a realistic perspective of what to expect in their future classrooms and provide practice 
with equipment they were most likely to encounter. At the beginning of the activity, the instructor introduced the 
equipment and explained the activity to the participants. Through this explanation, the participants were prompted to 
consider what they may already know about technology and more familiar equipment. An example provided was the 
consideration of setting up a television and a VCD/DVD player. In this description, the instructor explained the 
importance of taking into consideration factors such as signal direction, types of connectors (shape/size), and overall 
function of the equipment (i.e. what is the ultimate goal and what will happen if you successful). 

Once participants were actively engaged in the learning activity, rotation through the stations, the instructor 
served to facilitate group interactions. By freely rotating through stations, it was important to pay close attention to 
group conversations and monitor their actions and progress. If groups showed signs of struggle, the instructor would 
intervene and provide prompts to assist the groups to continue moving forward. Some of these questions included: 

• What is problem you are having? What doesn’t appear to be working correctly? 

• When you look at the labels on the equipment, are your “In” and “Out” connections correct? 

• Are you sure all the equipment is getting power? 

• If everything is connected correctly, what else could possibly be wrong? 

These questions, and others, often prompted the participants to reconsider their connections, check their 
power connections, turn on all the equipment, etc. (see Figure 2 and Figure 3). 




Figure 2 and Figure 3: Participants questioning their connections among equipment 



This typically led the groups into additional discussion eventually resulting in successful completion of the task. 
Participants were eventually able to question themselves in a similar manner as they progressed through each 
station, resulting in less reliance on the instructor and more reliance on other group members. 

Toward the end of the course, participants were given individual audio recorders to respond to a set of 
prompts regarding attitudes and reactions toward technology integration. The purpose was to capture possible 
changes in attitudes and reactions to things they had seen within their field placement classrooms and how that 
connected to prior experiences and what they had experienced throughout the technology methods course. A 
grounded analysis (Erlandson, Harris, Skipper, & Allen, 1993) was conducted by constantly comparing 
observations, audio recordings, and class deliverables. As a result, three prominent themes emerged as it became 
evident that a majority, if not all, participants voiced similar ideas and perceptions. 
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Data Analysis 



A pattern-matching logic (Yin, 2003), where empirically based patterns from the descriptive case are 
compared with predicted ones from theoretical propositions, was used to analyze data. Initially, theoretical 
propositions were derived from mental model and self-efficacy theories. If empirical and predicted patterns 
coincide, opportunity exists to strengthen the internal validity of interpretations. The particular type of pattern- 
matching used proposed rival explanations (Yin, 1994, p. 108). The important characteristic here is that each 
explanation involves a pattern of independent variables that is mutually exclusive with the other two. Finally, there 
was also room for simpler patterns to be revealed by the case. The goal, in the end, is to determine the best ways of 
contrasting differences and to develop theoretically significant explanations for different outcomes (Yin, 1994, p. 
110 ). 



Results 

Evidence exists to demonstrate not only growth regarding levels of confidence but also a deeper 
understanding of and appreciation for the educational possibilities of technology. Participants responded to open- 
ended questions regarding positive changes in attitudes, what they had discovered, and what they needed to become 
more confident. Most apparent were statements about realizing new possibilities, understanding how important it is 
to just use the equipment, and coming to realize what was needed in the future to enhance what was learned (see 
Table 2 for a summary of these findings). These results are interesting in light of previous work that identifies these 
concerns as prominent among first-year inservice teachers (Strudler, McKinney, Jones, & Quinn, 1999). 

Increase in Attitudes and Level of Comfort 

As participant change in confidence and self-efficacy was the primary motivation behind the design of 
instruction for this course, we carefully analyzed audio and video recordings for statements of perceived 
improvement or lack thereof. Five participants made it clear that they began the course with a high level of 
confidence in using technology for their teaching and were therefore unable to identify a major change. The 
remaining nine participants, however, were able to comfortably identify and admit that attitudes and level of comfort 
had increased as a result of instruction. Marcus, a traditional undergraduate, was particularly excited about 
describing what he had learned through his verbal response as follows: 

. . .these things [technology] were brought in; I had never used them myself actually, as far as hooking them 
up, and not had much use with them at all with display. I really got the feeling that - okay - 1 can look at 
these things. I can hook them up. And, you know, if it’s not meant to go together, it’s not going to go 
together so I’m not going to break anything. 

Katherine, a traditional undergraduate, noted excitement about the major change in her comfort level. She 
stated that in order to learn and become more comfortable, instructional activities need to “involve using something 
you’ve never used - just give opportunities to explore. ...” Jamie, another female, undergraduate student who had 
shown major improvement in her levels of confidence, responded: “My comfort level towards using technology has 
definitely increased. . .we’ve done a lot of activities where we put together the technology and practiced using it.” 

She further discussed the importance of having the opportunity to practice with technology, increasing confidence 
and desire to use technology in lesson plans. Christine, a traditional, undergraduate student, articulates her 
realization of the effects on her self-efficacy through written reflection as follows: 

After taking this Technology course I feel much more competent in the field of technology. Before taking 
this class, I thought knowing how to use Word and PowerPoint and being able to hook up equipment was 
all I needed to know. This class has shown me many different aspects of technology I never knew existed, 
which I can use in my classroom. I feel like I know a lot more about how to navigate different programs 
and even different computers. I now have many more resources to use to help bring technology into my 
classroom and feel comfortable using them. I no longer feel I am going to break the equipment. I know I 
can play around with equipment or programs until I figures out what I am attempting to do. Before, I never 
would have considered working with some of the technology we learned about in class. Now, I can’t wait to 
add it to my classroom (as cited in Combs & Evans, 2007, p. 2523). 
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Figure 4\ Participants completing the task, questioning their procedures 



Importance of Technology Within Teaching 

While activities for the course were designed to increase levels of self-efficacy, video and audio evidence 
showed effect on participant perceptions on realizing how limitless their teaching possibilities were with using 
technology for their lessons and how important it is to their teaching. Jayna, a traditional undergraduate, stated in her 
verbal reflection: 

My level of comfort toward incorporating technology in the lessons has changed a little bit, not a whole lot 
because I feel like I was pretty comfortable coming in, with some technology like Power Point, Microsoft 
Word, things like that. But I believe that if I want to be a good teacher, I think that I need to incorporate 
more technology into my lessons so that I can be up to date so that the children can see that there is more 
than just the standard - what their parents learned by is not always the best, so we shouldn’t try to limit our 
teaching by doing what was always done. 

Theresa noticed: “. . .the practice throughout the entire semester has helped me really understand there’s a 
lot more out there that I didn’t know about. . .”, but also pointed out, . .there’s a challenge to make it 
meaningful. . .the students need to be taught how they can make it meaningful for themselves. . .” Kelly, a traditional 
undergraduate, recognized that the class was a safe environment in which to “explore things that I may not have 
seen elsewhere. . .the more I know, the better I feel.” In reviewing participant responses, a majority responded with 
similar comments and clearly recognized the educational potential of the technology available in field placement 
schools and schools in which they plan to teach. 

Reaching Their Full Potential 

Even with careful planning and execution of lesson plans, participants were still able to identify they had 
not reached their full potential and their confidence levels could still improve. Many responses included a discussion 
of what students felt they needed in future lessons, courses or professional development activities. It was clear they 
had moved from thinking currently, regarding just what was necessary to get through the course, to thinking about 
their futures as educators. The main response from participants was that they felt more practice and opportunities 
were necessary to continue learning about technology integration. Jayna, a non-traditional graduate, summed up her 
reactions with the statement, “Practice makes perfect - just getting in there and doing what I can with technology 
will help to increase my confidence.” She realized that her learning must go beyond the doors of the classroom and 
to maintain what she already learned, she will need more practice. Marcus also mentioned the need for practice. He 
stated, “. . .it’ s just a matter of doing it and practicing.” It’s obvious, from observation and listening to her responses 
that Becky, a traditional undergraduate, still feels nervous. However, she is confident in stating, “I’m still a little bit 
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nervous. . .you never know if it [technology] will work or not. . .the more you use it, the more comfortable you’ll be 
with it and can create your own ideas when you are comfortable.” She also noted an increase in confidence and 
understanding of the need for teachers to be more educated. To illustrate, she wrote the following in her final 
reflection: 

After this course I do feel more competent with technology. I have learned a great deal not only with using 
technology, but along with numerous resources. I do feel more comfortable as well. I feel I can try to use 
equipment and see how it works. I also know I could try and look it up on the Internet and could find 
something about it one there. I think teachers need to be more educated on the options with technology. 
Technology is growing and students love to use the web. They seem to want to learn more when a 
computer is involved. I think teachers are hesitant with using technology in the classroom because they are 
unsure of how to use it themselves, (as cited in Combs & Evans, 2007, p. 2524) 

A summarizing comment from another student helps illustrate the overall impact of instructional strategies 
designed to encourage students in their practice and application along with supporting them in their personal 
attempts at learning and becoming more confident. Marcus, as highlighted earlier in the text, shares his personal 
insights regarding changes throughout the course and how the training he has received has affected his overall 
performance and attitudes. He states: 

I can tell you that honestly, I have always been uncomfortable with using technology in the classroom. It 
seems that while I am trying to teach a lesson it's hard for me to keep up with all of the gadgets or pushing 
buttons a lot. . .many times if I'm using a PowerPoint it's hard for me to stay on task with the PowerPoint to 
what part of the lesson I’m teaching. So this is something that, in going into the course of technology, that I 
was really hoping to benefit and improve on. 

I've gotten a lot better with keeping up with the slides and where I need to be, multitasking while using 
technology, keeping an eye on students, this is really actually kind of becoming a strength to me when it 
used to be something that was just weighing me down. So, yes, I feel much better about using technology 
in my lessons due to the chances that I’ve had to use it in classes and in teaching. 

The activities I feel were covered pretty well in my technology class because I’ve started using things and, 
to me, at this point, it's just a matter of doing it and practicing. The more I do it, the more comfortable I get 
and even my cooperating teacher has brought to my attention that he sees that I’m getting better with 
directing the class through the use of technology and - through direct use - me directing myself, I guess. 
And, it's just a matter of practice for me and I’m getting a lot more comfortable with it and I’ve created 
lesson plans and I feel like I’m very creative because I’ve gotten used to using technology a little more and 
I hope to get better at it the more time goes on and I feel that I will, and I feel that I am. (as cited in Combs 
& Evans, 2007, p. 2524) 

As is evidenced through the reflections above, these students were affected in several different ways in 
regards to their views on technology use in the classroom and their perceived personal self-efficacy (see Table 2 for 
a summary). While some addressed technology as a whole, both through and outside of course activities, several 
addressed specific topics within the course as being an immediate factor in changes to both their confidence and 
competence. The themes noted throughout provide evidence that as self-efficacy increases, learners begin to assign a 
higher value to what they are learning and realize the impact and effectiveness of not only using technology in their 
classrooms, but also what it would mean to continue to learn and improve on their skills and knowledge. 



Table 2\ Instructional strategies and their influence on self-efficacy 
Instructional Strategy Influence on Perceived Self-Efficacy 



Just-in-time training 



Helped participants realize their potential for current instructional 
planning and increased motivation for immediate integration and transfer 
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Providing a realistic perspective; 
using simpler technologies most 
readily available 


Gave participants chances to explore familiar equipment with 
opportunities to practice in a safe environment; slight mastery provided 
motivation for transfer and integration 


Activate mental model 


Allowed participants to place themselves in a familiar perspective and use 
what they already know; personal comparison introduced a perspective 
above the novice level providing an additional level of motivation and 
confidence 


Prompts/questions from instructor 


Helped participants activate existing mental models and reevaluate their 
position within the activity; prompted discussion and further questioning 
beyond scope of the activity which they answered confidently 


Assignment: writing instructions 


Participants were led to not only verbalize their understanding but 
compose their position in written form, allowing an extra level of 
processing and understanding; instructions can be used to provide self- 
assurance in future attempts and revised as experience increases 



Discussion 

This paper presents evidence to support a focus on the positive influences on self-efficacy through the 
design of sound instructional strategies. Pre- service teachers who participated in the research study provided verbal 
and visual evidence of potential as teachers when using technology for classroom instruction. Many participants 
concluded that without a course designed to demonstrate new techniques, given hands-on experiences, and increased 
self-efficacy, they would not have felt comfortable using technology in the classroom. The course and activities 
throughout allowed students to practice skills and find new ways to use common technologies. A majority of 
participants were confident in their reactions to sharing what they had learned about technology, its role in the 
classroom and how their attitudes had changed. In addition, students were much more able to verbalize their need 
for future training, the importance of practice and what they understand will need to happen for confidence to 
increase. These changes in attitude bode well for the formation of “collaborative, subject specific technology inquiry 
groups” that Hughes (2005, p.277) proposes for future teacher professional development. 

Admittedly, limitations of the current study exist. The reporting of perceptions cannot guarantee 
implementation in the classroom. Consequently, our plan is to follow-up with additional research participants and 
monitor them throughout their field placements to identify additional themes and help determine the actual degree of 
action on words. Pope, Hare, and Howard (2005) have noted the positive effects appropriate modeling has on pre- 
service self-efficacy toward technology integration. Finally, we recognize that data collection in the field is fraught 
with unexpected outcomes - one of them being that participants may become frustrated or misunderstand the 
purpose of research; an obligation as qualitative researchers is to ensure that participants are treated ethically and 
with due respect. We recognize this issue and explain how we dealt with it in separate work (Evans & Combs, 

2007). 



Conclusion 

In summary, instructional designers and technologists are at the forefront of promoting a culture of ubiquity 
when it comes to technological resources in the classroom (cf. Evans & Powell, 2007). Consequently, we have an 
inescapable duty to ensure that pre-service teachers are prepared in knowledge, skills, and affect to effectively and 
creatively adopt and use these technologies - be they computers, projectors, digital video recorders or the like. The 
question is this: “What instructional strategies are most effective to promote technology integration in the middle 
school classrooms of tomorrow?” Researchers will continue to propose similar questions and conduct additional 
research to help answer this question. At this point, the continuation of research can only encourage those in practice 
to continue or discontinue their current approaches to preparing teachers to integrate and teach with technology. 

With this research we call for theory- and evidence-based decisions to guide teacher preparation. 
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Abstract 

This study examined how instructional design (ID) experts used their prior knowledge and previous experiences to 
solve an ill- structured instructional design problem. Seven experienced designers used a think-aloud procedure to 
articulate their problem-solving processes while reading a case narrative. Results, presented in the form of four 
assertions, showed that experts 1) narrowed the problem space by identifying key design challenges, 2) used an 
amalgamation of knowledge and experience to interpret the problem situation, 3) incorporated a mental model of the 
ID process in their problem analyses, and 4) came to similar conclusions about how to respond to the situation, 
despite differences in their initial conceptualizations. Implications for the education of novice instructional designers 
are discussed. 

Experts in any field tend to be better problem solvers than novices. Faced with a problem situation, experts 
quickly form solutions that are more likely to be effective than solutions formed by novices. This characteristic of 
expertise has been documented in diverse domains: playing bridge (Chamess, 1979), solving physics problems (Chi, 
Feltovich, & Glaser, 1981), repairing electrical generators (Jacobson, 1988), driving a taxi (Chase, 1983), and 
revising instructional text (LeMaistre, 1998). Experts are able to achieve this superior problem-solving performance, 
in large part, because they have at their disposals vast, well-organized stores of domain-specific knowledge, gained 
through extensive experience (Bruer, 1993; Glaser & Chi, 1988). Expertise is comprised of two major components: 
abstract knowledge and practical experiences (Bonner, 2007; Laurillard, 2002). While expert knowledge may arise 
from only one of these sources, both may be necessary to develop the kind of “fluid expertise” (Bereiter & 
Scardamalia, 1993) that allows individuals to adapt and apply their knowledge to novel situations. According to 
Bereiter and Scardamalia, practical experience by itself may lead to a kind of “crystallized expertise” that results in 
individuals who may not be particularly good problem solvers because they simply implement well-practiced 
procedures. In contrast, individuals with fluid expertise use both abstract knowledge and practical experience to 
think through a problem in a more dynamic and constructive way. While crystallized expertise would include the 
acquisition of automaticity after extensive practice in a relatively stable and constant system, fluid expertise would 
be characterized by flexibility that is responsive to changes in a dynamic world (Feltovich, Spiro, & Coulson, 1997). 

Drawing on the expert-novice literature, Ertmer and Stepich (2005) outlined six dimensions that 
characterize the problem solving processes of expert instructional designers. Central among these dimensions is the 
expert’s ability to “synthesize” a particular problem situation; that is, to formulate a clear, coherent representation in 
terms of one or two central issues. This kind of synthesizing has been a recognized aspect of expert practice for a 
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long time. For example, Larkin, McDermott, Simon, and Simon (1980) found that physics experts routinely 
generated a physical representation of the problem situation before attempting a computational solution. More 
specifically, when experts were given complex problems to solve, they frequently began by drawing a sketch of the 
central elements of the problem. This physical representation reduced the problem space, allowing the experts to 
identify relevant variables and test relevant qualitative hypotheses. Once these hypotheses were checked, the experts 
would fill in the details and solve the problem quantitatively. In the nursing field, Benner (1984) used the term 
“recognitional ability” to describe the ability of expert nurses to discriminate relevant from irrelevant information 
and to develop a context-dependent, holistic perceptual understanding of the patient situation. According to Benner, 
this ability is based on collected practical experiences that expert nurses use to continually refine their abstract 
knowledge. In both cases, the authors suggested that experts begin the problem solving process by developing a “big 
picture” understanding of the problem situation. 

Similar results have been shown within the field of instructional design. For example, Perez and Emery 
(1995) asked expert and novice instructional designers to design a computer simulation on diesel engine mechanics. 
They found that experts were more likely to identify a central element of the problem (e.g., the characteristics of the 
target audience) and to return to this element as they began to work out the design details. Perez and Emery referred 
to this approach as a “breadth- first, top-down, progressive” design strategy. In another study involving the revision 
of instructional text (LeMaistre, 1998), the expert instructional designer initially identified the lack of overall 
structure of the text as the primary problem and continually referred to the importance of structure throughout the 
revision process. LeMaistre noted that the expert was explicit in the creation of the problem space and employed 
“strategies of constantly adjusting decisions and decomposing the problem into manageable parts” (p. 31) so that 
related aspects of the problem could be addressed collectively rather than in an isolated fashion. 

While it seems clear that experts engage in the kind of synthesizing described above, it is less clear how 
this process is influenced by the experts’ vast stores of knowledge. Building on the idea that expert knowledge is 
made up of abstract knowledge and practical experience, one view is that synthesizing is based, primarily, on the 
recall of abstract knowledge. In this view, experts define a problem in terms of conceptual principles drawn from 
their stores of domain knowledge (Ertmer & Stepich, 2005). For example, Chi et al. (1981) asked experts and 
novices to outline solutions to physics problems. While novices typically defined the problem in terms of literal 
objects and terminology used in the problem statement, experts were more likely to identify a “second order 
feature,” referring to a feature that was not explicitly described in the problem statement, but that was derived from a 
small piece of given information that activated a relevant schema in the experts’ existing knowledge. Glaser and Chi 
(1988) noted that, with experience, experts encode not only the procedures for solving relevant problems but also the 
conditions under which they are applied. 

An alternative view is that synthesizing is based, primarily, on the recall of practical experiences. In this 
view, experts define a problem in terms of a similar situation drawn from prior experiences (i.e., a case). For 
example, Rowland (1992) found that instructional design experts typically associated a given situation with similar 
problems they had previously encountered and used those prior experiences to develop an initial picture of the 
current problem and how it might be solved. Similarly, Perez, Jacobson, and Emery (1995) observed that 
instructional design experts often reflected on past design problems and solutions and compared them with the 
problems at hand. Klein and Calderwood (1988) studied decision making among urban fire commanders, wild land 
incident commanders, and tank platoon commanders and found that these individuals based their decision making 
more on prior cases than abstract principles. This interpretation is supported further by research on case-based 
reasoning (Kolodner, 1997), which posits that experts have amassed rich libraries of case experiences that they 
apply, through a type of analogical reasoning, when solving new problems. 

To elaborate, case-based reasoning is defined as “solving a new problem by remembering a previous 
similar situation and by reusing information and knowledge of that situation” (Aamodt & Plaza, 1996, p. 40). 
Theorists claim that human reasoning is case-based; that is, we all have different experiences that are stored in our 
memories and then reused when new problems trigger the recall of similar situations (Kolodner, 1993; Schank, 

1999). Jonassen and Hemandez-Serrano (2002), after reviewing studies in multiple contexts, proposed that “experts 
relied more heavily on cases based on past experience than on abstract principles when making decisions with a high 
degree of uncertainty” (p. 68). They argued that cases and stories work more effectively than abstract rules or 
principles in knowledge construction because they “require less cognitive effort than exposition” (p. 66). Moreover, 
stories or cases facilitate vicarious learning by providing a substitute for first-hand experience (Jonassen, 1999). 

Although these two explanations of synthesizing appear conflicting, they may not be mutually exclusive. 
For example, Genberg (1992) suggested that expertise might be viewed from two different lenses: 1) an information- 
processing lens and 2) an intuitive lens. While the former emphasizes the organization of knowledge and the 
progression of skill acquisition, the latter focuses on the relevance of past experiences in a particular context. 
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Kolodner and Guzdial (1999), while strong advocates of reasoning from cases, stressed that abstraction is necessary 
for organizing, or indexing, cases within one’s library, as well as for efficient retrieval. In other words, experts seem 
to extract guidelines and principles from concrete experiences that they then apply to new problem solving 
situations. This idea is supported further by research by Didierjean and Cauzinille-Marmeche (1998) who 
demonstrated that individuals can develop, and even use simultaneously, these two types of reasoning processes. 

Purpose 



The purpose of this study was to determine if instructional design (ID) experts synthesized the issues 
presented in an ill- structured problem scenario (as described in the literature), and if so, how abstract knowledge and 
practical experience were used during the synthesizing process. Based on the six dimensions of expert thinking 
described by Ertmer and Stepich (2005), we selected synthesizing, a critical characteristic that distinguishes experts 
from novices, to further our understanding of how experts use their prior knowledge of rules and principles, and/or 
draw upon their previous experiences. 



Method 



Overview 

This study was designed to examine the processes that experienced instructional designers use when 
solving ill-defined instructional design problems. Data consisted of a demographic survey, think-aloud protocols, 
and interviews with seven participants. The think-aloud protocols captured experts’ verbalizations during the 
problem-solving process and the interviews gathered additional data about how experience and knowledge were 
used during the process. 

Theoretical Framework 

We used grounded theory, a method of qualitative inquiry designed to generate an explanatory theory of a 
specific process or phenomenon (Glaser & Strauss, 1967; Strauss & Corbin, 1997). Grounded theory is an inductive 
approach in which theory is derived from the data through a process of asking questions and making comparisons. 
The primary objective is to expand upon an explanation of a phenomenon by identifying the key elements and the 
relationships among them within the specific context of the research study (Davidson, 2002). Thus, in this study, a 
grounded theory approach enabled us to develop a theoretical account of the characteristics of expert problem 
solving while simultaneously grounding the account in empirical data (Strauss & Corbin, 1997). 

Role of Researchers 

This study was designed and implemented by a research team consisting of six doctoral students and one 
faculty member at a large mid-western university. A second faculty member, located at a large western university, 
acted as a consultant to the team during the design and implementation of the study. All but one student had 
previously completed an advanced instructional design course, which utilized ill-defined case problems. Students 
had a range of previous ID experiences in both educational and business contexts. 

Prior to the start of the study, a pilot was conducted with one participant, during which the entire research 
team observed and/or participated in implementing the data collection procedures. Subsequently, the research team 
divided into two subgroups with each group taking primary responsibility for conducting the research with three of 
the six remaining participants. As a team, students worked to define the research protocol, to modify specific data 
collection procedures that were either problematic or unclear during the pilot, and to clarify each person’s role in the 
subgroups. The researchers carefully checked and monitored each other during the entire research process, 
reviewing transcriptions for accuracy and requiring clear evidence of initial interpretations. For example, specific 
claims were linked to supporting data (using a line-numbering system for each transcription), thus enabling team 
members to challenge or support initial interpretations and to provide either additional or counter evidence. 

Participants 

Seven expert instructional designers (four women and three men) were purposefully selected. Each 
participant had eight or more years of instructional design experience, in a variety of settings. Demographic data 
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(e.g., years and types of experience; current positions and responsibilities) were collected via a short online survey. 
On average, participants had 20.5 years of instructional design experience, ranging from 8 to 32 years. Four 
participants were currently working in higher education (with two of these holding a Ph. D. degree), while three 
were employed in the business sector. All seven designers indicated that they regularly participated in all aspects of 
the instructional design process (analysis, design, development, implementation, evaluation). In addition, six of the 
seven participants indicated that they had previous experience with face-to-face, computer-based, online, self- 
instructional, and hybrid instructional delivery formats. 

Data Collection 

Following the completion of the demographic survey, participants met individually with one of the two 
research subgroups to participate in a think-aloud process in which they read and reflected on an ill-defined 
instructional design problem. The problem (Hooper & Doering, 2007) was presented via a 12 page, double-spaced, 
typed narrative and dealt with the topic of converting a face-to-face HIV/AIDS workshop to an online format — a 
topic for which none of the participants had previously developed instruction. Each data collection session lasted 
about two hours and included a warm-up exercise, the think-aloud procedure, and a retrospective interview. The 
warm-up exercise included a small problem that the participants used to practice the think-aloud approach (e.g., 
identify the number of windows in your house). Retrospective interviewing (Ericsson & Simon, 1980) occurred 
immediately after the think aloud as a way to help participants reflect on, and verbalize, their thought processes 
during the think aloud, drawing from both long- and short-term memory (e.g., Describe the method you used to 
determine the number of windows in your house.). In addition, interviews included questions to clarify comments 
participants made during the process and to explicate how knowledge and experiences were used. The interviews 
included questions such as, “What was the first thing you thought about as you read the case?” “What made you 
think of that?” or “Where did you learn that?” All sessions were videotaped and transcribed. 

Data Analysis 

Transcriptions were examined using a constant comparison method, with specific attention given to 
participants’ references to prior knowledge and experiences. Initially, each researcher conducted an analysis of a 
single transcription, looking for evidence that related to our two research questions, but without establishing 
preconceived ideas about what might be discovered. This resulted in a set of tentative profiles that captured each 
participant’s response to the case situation. Following this, two researchers applied a modified open-coding process 
using an electronic copy of each transcription, inserting comments and highlighting quotes that seemed particularly 
relevant to our questions. However, rather than create a set of categories and subcategories as is typical in open 
coding (Creswell, 2003), we created a set of themes that reflected each participant’s responses. Themes for each 
participant were shared and discussed between the two researchers as they were developed. During these 
conversations, no attempts were made to come to consensus, but merely to note the similarities among the themes 
and to seek clarification and additional evidence if the interpretations were unique or unusual. 

After creating themes for each participant, the researchers then looked for similarities across participants as 
the first step in creating a set of assertions that could be applied to the majority, if not all, of our participants. If the 
themes were not evident among at least four participants, they were not used in the final set of assertions. Finally, 
after the assertions were developed, they were presented to the rest of the research team (with evidence) for 
additional comment and final team verification. The team then worked together to find relevant supporting or 
contradictory evidence from the literature. 

V alidity/Reliability 

Lincoln and Guba (1985) recommended that qualitative results be evaluated using the standard of 
“trustworthiness,” as established by credibility and confirmability. In this study, credibility was gained though 
triangulation of multiple data sources including a demographic survey, think-aloud protocol, and retrospective 
interview. The use of multiple researchers led to confirmability of the data. Throughout the study, weekly meetings 
of the researchers helped to ensure understanding of our research questions, consistency of data collection, and 
interpretations of data. Data analysis involved both individual and collaborative efforts in order to develop and 
verify the emergent themes and overall assertions. A line numbering system was used to identify specific data 
supporting each finding, thus creating a traceable link between evidence and assertions. Finally, participants verified 
our interpretations, immediately during the interviews, and later through their responses to specific email queries. 
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Results and Discussion 



In this study we asked questions about how experts used their knowledge and experience to solve ill- 
structured problems presented via a case study narrative. We present our findings in the form of four assertions that 
were developed through our analysis and interpretation of the data. Excerpts from the participants’ think- aloud and 
interview protocols are integrated with interpretive commentary to support each assertion. Subsequently, we discuss 
how each assertion is, or is not, supported by the literature. 

Assertion 1. In identifying the key design challenges, experts narrowed the problem space. For some, 
this seemed to occur either in place of, or prior to, making a synthesis statement. According to the literature, a major 
task of the problem-solving process involves “being able to articulate a clear and concise representation of the 
problem(s) in a particular situation” (Ertmer & Stepich, 2005, p. 39). Referring to this as the ability to “synthesize” 
the issues in a problem situation, the authors noted that this is a key characteristic of expert instructional designers. 

In this study, all seven participants articulated specific design challenges related to converting a highly 
interactive face-to-face workshop to an online format. Specific aspects of the case situation were highlighted either 
because they were seen as particularly central to the effectiveness of the workshop and/or particularly difficult to 
translate to an online environment. For example, as Jacob read the sentence, “Experiences were extremely powerful” 
he stopped and added “which is going to be wonderfully fun to try to do with the Internet.” Similarly, when Marlene 
read the sentence, “The face-to-face . . . workshop was presented to approximately 50-60 participants who traveled to 
a single location and met for approximately 1 6 hours over a two-day period,” she noted, “That seems problematic to 
me, for something that’s online.” Additional challenges highlighted by the participants included, among others: the 
importance of interactions among the workshop participants, learning by doing, resolving access issues, facilitating 
and maintaining behavior change advocated by the workshop leaders, and “smoothing out” interpersonal 
relationships among the designers in the case (described in more detail in Assertion 4). 

Among our seven participants, “highlighting” seemed to be more common than “synthesizing.” Despite the 
fact that every participant articulated specific challenges related to converting the face-to-face workshop to an online 
format, not every participant synthesized these challenges into a clear concise statement, as hypothesized earlier. 
While Jacob, for example, stated the problem in a very straightforward manner (“I see the problem as determining 
what are the essential characteristics to changing behavior?”), others simply pointed out a number of different 
elements that would be difficult to transfer to the online environment without making a direct statement about how 
these elements contributed to, or comprised, the core problem. For example, as Jill engaged in the think-aloud 
process, she stopped periodically to comment on new challenges as they were introduced in the case narrative. These 
challenges related to a wide variety of issues: putting an interactive workshop online, moderating discussions, 
providing appropriate counseling services, securing online permissions, resolving access issues, and limiting the 
amount of time required. Jill noted that she usually begins the design process by “looking for some sort of initial 
analysis, starting with some kind of objectives” but because these were not immediately available, she described her 
approach as just “kind of reacting to things as we went through.” This lack of a synthesis statement, then, may have 
been due to a stylistic difference among participants or to a variation in the way the researchers asked the 
participants to state the problem. Alternatively, it may suggest that synthesizing does not always result in a single 
concise statement of the problem. Instead, as Gredler (2004) suggested, experts may identify key information within 
a situation and use that information to create a mental map of the problem. 

In almost all instances in which synthesizing occurred, it was preceded by highlighting. Most often, the 
synthesis statement captured one or two key design challenges that the participants had highlighted during the think- 
aloud process. For example, after highlighting a number of problem elements, Simone stated: 

If that data is [sic] actually true and correct, I’d want to hone in on what elements of instruction itself — the 
format, the instruction, the affective aspect of it, group interaction [all problem elements noted earlier] — all 
of that and say, “What part of this made it the most successful?” and “Can this be duplicated?” 

As another example, after mentioning a number of potential problem elements, Thad synthesized the issues 
with the following statement: 

Soon you’re going to have some kind of table [in which] you compare what they’ve done in the past, and if 
it worked then, are you going to be able to transfer that over into the online environment and help them to 
solve some problems they are having in the transfer? That’s probably where they are having their most 
difficulty in creating this thing, is that transfer of some of these things. 

The literature suggests that experts tend to translate ill-defined or unfamiliar problems into well-defined or 
more familiar problems as a way to narrow the problem space and search for a solution (Glaser & Chi, 1988; 
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LeMaistre, 1998; Perez et al., 1995). According to Rowland (1992), expert designers approach new problem 
situations using existing “frames of reference,” built from their previous knowledge and experiences. Frames of 
reference, then, may be one way in which experts make this translation from an ill-defined problem to a more 
defined one. Similar to what Rowland (1992) and Perez and Niederman (1992) found, each participant in this study 
understood the case problem in terms of the personal experiences and perspectives s/he brought to the case as much 
as by the information provided by the case narrative. For example, based on her 22 years of experience as a designer 
in business and industry, Marlene brought a “training” frame of reference to the problem: “What is the problem? 

Will training address it? Who are we dealing with?” She elaborated on her specific perspective: 

I look at it from a training perspective and then I add all of the human baggage that goes with it, or that is 
going on there. I go to “what is the target audience” and get a clear definition of what that is. And then [I 
get] a clear definition of what the training issues are by defining the objectives. 

Sean, who had a background in counseling psychology used a “consulting” perspective: “The problem is 
more about the interaction of the cultural differences between the organizations as much as it’s about a lack of 
specificity of the goals.” As he explained further: 

This is a very typical problem ... if s a challenge for consultants who work with the military, for example. 
And I was sort of using that as one of my frames of reference. I have experience in that. I do my research 
with consultants as well and . . . there is always tension between research design and product design. 

In contrast, Sammie used her communication background to frame the problem in terms of the type of topic 
that needed to be addressed and the kinds of conversations that needed to occur: “With such a . . . touchy topic, I 
would want to have control in the room so if something happened I would be there to handle it. ... I have a 
background in communications and all of my teaching is done in very small groups where I can watch the faces of 
my students and especially their non-verbal behavior and make instant corrections in the instruction.” 

As a final example, Jacob, who was in charge of instructional computing on a large university campus, 
used an administrative perspective to consider whether the “real” problem could be addressed in an online 
environment: “I pull from components of projects and they are kind of reorganized into other types of situations. I’m 
primarily an administrator, so my categories right now are: project timelines, budgets, legal, those types of things.” 
In summary, participants in this study all identified multiple design challenges in the case scenario, with 
these challenges later being combined, by five of the seven participants, into the articulation of a synthesis statement 
during the interview process. As suggested by Schon (1983), the first step in the problem solving process is problem 
finding or problem setting: “The designer must make sense of an uncertain situation that initially makes no sense” 

(p. 74). As part of this process, expert designers identify the constraints, or problem elements, of the given situation. 
Goel and Pirolli (1992) suggested that during this process, expert designers may explicitly try to change the problem 
situation so it more closely matches their personal expertise and knowledge. Although we didn’t observe our 
participants use their frames of reference to actually transform the case information, they were employed to filter 
through the details, facilitating a focus on those case details that were judged most critical. 

Assertion 2. When analyzing ill- structured problems presented via a case narrative, instructional design 
experts used an amalgamation of knowledge and experience. While all of our participants referred to specific prior 
experiences and six of seven referred to some specific piece of abstract, academic knowledge (e.g., ADDIE model, 
Gagne’s types of learning, message design), their interpretations of the case details relied primarily on an 
amalgamation, or blend, of knowledge and experience. When asked, specifically, how they knew to use a particular 
strategy or to consider particular issues, our participants typically referred to previous experiences. For example, 
Marlene responded, ““How do I know that? I know that just from my experiences working in this company,” while 
Simone reflected, “I’m thinking of my own experiences with taking online courses.” Sean captured the general 
feeling of all of our participants when he stated, “I have been in that situation many times.” 

In general, the recollection of specific ID experiences or specific ID principles was rare. Furthermore, if 
participants recalled specific experiences, they didn’t really use the information from that experience to narrow the 
problem space or solve the problem (at least not in any obvious way). If they recalled specific book knowledge, it 
was usually embedded within their recall of experiences. More typically, the participants recalled a “mix” or “blend” 
of experiences that were relevant to the current problem. Consider the following quotes from three participants: 

• “I can’t say, oh this really reminds me of this, but there is . . . all the little pieces remind me of 
something.” (Jill) 

• “I’m thinking it was background experiences but there isn’t one specific one.” (Thad) 

• “I have a blend in my head that is from many different experiences ... in my mind, it just all blends 
together ... I pull from components of projects, and they are kind of re-categorized into other types of 
situations.” (Jacob) 
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Rather than accessing a single specific prior experience (a case), participants extracted from their collected 
experiences one or more relevant rules, which they applied to the current situation. These rules were not ones that 
could be found in an ID textbook, but were much more idiosyncratic and were drawn from the unique collection of 
previous experiences that each participant brought to the current situation (Klein & Calderwood, 1988; Kolodner & 
Guzdial, 1999). According to Davenport and Prusak (2000), rules of thumb are “guides to action” that have 
developed over time through extensive experience and observation (p. 10). Schank (cited in Davenport & Prusak) 
referred to these internalized responses as “scripts,” which, like play scripts or computer programming codes, act as 
efficient guides to complex situations, offering plausible routes through a maze of alternative solutions (p. 11). 

As an example of how our participants applied rules of thumb during their analyses of the case narrative, 
Thad suggested that before you can decide what to translate to an online environment you need to know exactly 
what’s making the current materials effective. This rule is captured in his comment: “You have to take it [the 
workshop] apart and make sure that the type of thing isn’t going on where you think it is effective, but yet it really 
wasn’t teaching what they needed.” As another example, Sammie used a set of rules to guide her decision-making 
that related to her background in communication. These rules helped her decide how to handle sensitive topics in a 
workshop environment: “If [the workshop involves] interaction and the subject is controversial, the delivery should 
be face-to-face.” For Sammie, the decision to keep the workshop face-to-face appeared to be based on what she 
viewed as a critical rule of thumb. Similarly, a related rule was captured when she stated, “If it’s a run-of-the-mill 
topic, then it could be successfully converted to the Internet.” 

In summary, experts in this study used an amalgamation of knowledge and experience to analyze the 
problem situation presented in the case narrative. Similar to what Davenport and Prusak (2000) described as a “fluid 
mix of framed experience” (p. 5), our participants appeared to access domain-specific knowledge, which was built of 
experience. As noted by Kolodner (1988; 1997), expert problem solvers tend to access their case memories multiple 
times during a problem-solving episode, thus allowing them to recall several cases (previous experiences), rather 
than just one, to be used during the process. Results from this study support this contention. 

Assertion 3. When recalling previous ID knowledge and experiences to solve an ill-structured problem, ID 
experts accessed a mental model of the instructional design process to guide their thinking. According to the 
literature, experts have a large store of organized domain- specific knowledge (Bransford, Brown, & Cocking, 2000; 
Bruer, 1993; Glaser & Chi, 1988), organized as schemas or deep knowledge structures (Bedard & Chi, 1992; 

Brophy, Hodge, & Bransford, 2004). Johnson (1988) suggested that expert knowledge is organized schematically as 
a mental model of the relevant system. Experts use this mental model, then, to create a large-scale, qualitative 
representation of the current problem (Larkin et. al., 1980) as a first step in solving the problem. 

In this study, all seven of our participants made statements suggesting that they approached the case with 
some kind of instructional design process model in mind. The models varied, but everyone had one. Two types of 
models accounted for six of the seven experts in this study. For example, three of the participants used an “audience 
first” model: 

• “The first thing I like to do is write down what the target audience is.” (Marlene) 

• “You need to know what the audience is and the more you know about the audience the better you can 
design something.” (Sammie) 

• “OK, so I’m honing in on the target audience because any time you design instruction you are 
designing it for a particular audience and it is helpful to know who that audience is and as much 
information as possible about that audience.” (Simone) 

Three of the remaining four participants used an “outcomes first” model. This shows up, specifically, in 
comments about what is missing in the case: 

• “. . . a lot of the objectives are not clear to me. So, I’d really want to nail it down first of all.” (Jill) 

• “What are the objectives here? . . . These are goals, they’re not objectives - they’re way too broad.” 
(Jacob) 

• “As a designer, the challenge here is figuring out the complex combination of learning outcomes that 
they are after . . . thinking about a solution is way too far down the road because we don’t know what 
the goal is.” (Sean) 

Finally, one participant used a “domains of learning first” model. A critical step in Thad’s process was to 
“classify the type of learning that is occurring.” In contrast to other experts, Thad did not refer to objectives or goals. 
Instead, he referred to the type of learning, specifically questioning whether the focus of the workshop was on verbal 
information or attitudes and noting that “we’ve got to go at it a little different” depending on the type of learning 
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involved. This idea of domain-specific instructional strategies is a central principle in Gagne’s instructional design 
model, which suggests that Thad’s mental model was based on Gagne’s theory. 

In general, our participants used these mental models to guide their thinking about the case. Typically, they 
did not follow their models on a 1 to 1 basis, like a recipe. Instead, they used their models in a broader, more 
heuristic way. While the nature of the think-aloud task may have limited their uses of the model to the beginning 
stages of the analysis task, our participants appeared to apply them in two explicit ways: 1) to structure their 
searches for information, as illustrated by Marlene and Simone, respectively (“I needed to look for information to fit 
into a model like the ADDIE model.” “. . . The ADDIE Model is just sort of a nice little acronym that reminds me 
have you covered all of these bases. It may not be in that order necessarily in reality.”), and 2) to focus their 
attention on initial information considered critical to the instructional design process. This isn’t necessarily the first 
step in a particular textbook model, but rather the element of the ID process that the experts saw as particularly 
critical at the start of the process. For example: 

• Sammie (“audience first”) focused first on the needs of the target audience. 

• Jacob (“objectives first”) distinguished between objectives and goals and wanted to know what the 
objectives were. It’s worth noting that objectives are not the first consideration in the Dick and Carey 
model (the textbook model that Jacob mentioned), which supports the idea that the mental models were 
individual heuristics rather than textbook-based recipes. 

• Thad (“domains of learning first”) based his thoughts about selecting the instructional media on the type 
of learning involved. 

The fact that our designers all accessed a mental model of the ID process is not surprising, as it is supported 
by the literature. Recent research by Campbell, Schwier, and Kenny (2006) suggested that designers reference 
conventional ID processes in their conversations about design, although their practice varies significantly according 
to context. Perez and Neiderman (1992) also reported that the experts in their study used a design process that 
reflected a systems approach, yet varied in how they implemented the process. The authors argued that these 
differences were related to the frames of reference that the experts used when making their design decisions. Again, 
this may explain why our participants used the models that they did: that is, an “audience first” model was more 
compatible with a communications frame of reference, while an “outcomes first” model was more compatible with 
an administrator’s frame of reference. After participating in a large number of relevant projects, the mental models 
of the ID process enabled our experts to reason, not from textbook principles or models, but rather from “first 
principles” (Merrill, 2002; Reigeluth, 1997; Winn, 1997). This is similar to what other researchers have described: 
Experts tend to look past the surface details in a problem to focus on the underlying principles or “big ideas” 
embedded in the situation (Bransford et al., 2000: Glaser & Chi, 1988; Larkin et al., 1980). 

Mental models are one characteristic that have been used to distinguish between expert and novice 
designers. In general, novice designers have access, primarily, to textbook models, which don’t necessarily apply to 
novel problem situations (Atherton, 2002; Reimann & Schult, 1996). In contrast, based on their many years of 
experience in a variety of contexts, experts are able to recognize patterns of practice and generate solutions based on 
those that have worked in similar situations (Hardre, Ge, & Thomas, 2006; Kirschner, Sweller, & Clark, 2006). 

Assertion 4. The experts came to the same , or a very similar, conclusion about how to respond to the 
situation. Polya (cited in Wilson, 1997) stated that once we’ve figured out how to see a problem in a certain way, the 
solution becomes obvious (p. 23). Given that the frames of reference used by our participants all incorporated some 
variation of an ID process model, (Assertion 3) this may explain why (and how) the experts in our study all came to 
see the design problem in a similar way. As captured by Assertion 1 , all seven participants articulated explicit design 
challenges related to converting the face-to-face workshop to an online format. Specifically, as illustrated by Table 
1, the participants mainly focused on four related issues: the affective nature of the workshop, the use of diverse 
instructional strategies, achieving the goal of behavior change, and the interactive nature of the workshop. 



Table 1. 

Design Challenges Identified by Participants 



Identified design challenges 


Jacob Jill 


Marlene 


Sammie 


Simone Sean 


Thad 


The affective nature of the workshop (e.g., 
powerful experiences, intense engagement) 


X X 


X 


X 


X X 


X 


Converting the instructional strategies that 
were effective in the F2F workshop to an 
online format 


X 


X 


X 


X 


X 
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Achieving, measuring, and maintaining 
behavior change 


X 




X 




X 


X 


X 


The interactive nature of the workshop 


X 


X 


X 




X 






The length of the workshop 


X 


X 


X 










The interactions between the designers 
involved 








X 




X 


X 


Ethical and legal issues 


X 




X 











All seven designers highlighted the difficulty in recreating the affective nature of the workshop in an online 
environment. For example, Simone commented, “There are some things that cannot be [translated online]. If it is 
[just] information and that changes behavior, then yes, there are ways to get at that. But if it’s this closeness and a 
support group ... if it’s some other element; there are some things that just cannot be duplicated online.” Similarly, 
Jacob noted, “The empathy — yeah, that’s going to be a challenge . . . Their options are going to be very limited.” In 
fact, due to the affective nature of the topic and the methods used in the face-to-face environment (hot cognitions, 
sexually explicit media, etc.), several designers expressed a healthy skepticism toward making this conversion. 
Simone and Sammie, in particular, challenged the decision to translate the workshop. Simone said, “The first 
question I would ask is: Does this really lend itself to e-leaming? I am not sure. You know, you will have to 
convince me. Why are they even doing this? What is the overriding advantage over what they’ve got, which is 
already successful?” This led Simone to conclude “Maybe it shouldn’t be an online course at all.” Sammie echoed 
Simone’s concerns and quickly came to the conclusion that this conversion should not be made: “If the previous 
workshop has been successful, I would not go with the Internet ... it would open all sorts of cans of worms.” With 
such a . . . touchy topic, I would want to have control in the room ... I would stick with what’s been working.” 

In addition to outlining the challenges of translating affective content and methods to an online 
environment, five of the seven designers identified specific challenges related to implementing the diverse range of 
strategies that had been effective in the face-to-face environment noting, “We’ll have to think about ways for doing 
that” (Marlene). Simone commented specifically on the “learning by doing” approach favored in the face-to-face 
workshop, stating, “As an educational approach this is great, but how you do that online . . . that’s something to think 
about.” Five designers also identified challenges related to the goal of the workshop, that is, to create a change in 
behavior. For Jacob, particularly, this was the focal point of the case: “. . . [for] how long does that behavior change, 
and for which people . . . that’s what this group has to deal with.” 

The identification of these key design challenges, then, shaped how our participants envisioned addressing 
those challenges. Although participants conceptualized the design challenges in slightly different ways, initial 
strategies for addressing the design issues were very similar. That is, six of the seven participants described the need 
to determine the specific characteristics of the current workshop that were essential to success. For example, Thad 
noted the need to identify and translate the strategies that were successful in the face-to-face workshop, while Jacob 
proposed determining the characteristics of the workshop that were essential to changing the behavior of the 
participants, describing these in terms of must-haves vs. nice-to-haves. Similarly, Marlene discussed the need to 
determine which workshop components provided “value added.” 

The results of this study suggest that expert designers, despite some variation in their conceptualizations of 
the issues in the problem situation, still reach similar conclusions about how to address those issues. This finding is 
similar to that of Spector and Koszalka (2004) who found that experts conceptualize complex problems in 
recognizably similar ways. As pointed out by Driscoll and Carliner (2005), this may relate to the nature of the 
problem-finding process in that early problem identification subsequently limits the number of solution paths that 
can be considered. It is conceivable that by initially highlighting similar elements, our designers had, in effect, 
constrained the number of possible solution paths. However, this finding contrasts with what Rowland (1992) 
reported in his study, in which the experts were described as demonstrating a “significant amount of variation” (p. 
81), both in terms of their problem representations and proposed solutions. Rowland attributed this result to the 
different frames of reference used by his participants. While no two designers in our study appeared to use the same 
frame of reference when analyzing the problem, they all incorporated key elements of the design process within 
those frames, a similarity that was not described by Rowland. Perhaps, the “significant” variation in initial problem 
representation left more solution paths open for Rowland’s participants to explore. Finally, we must also consider 
the simple explanation that the case narrative used in this study was not as complex as the problem that Rowland’s 
experts encountered, thus allowing our experts to more readily represent and address the core issues in similar ways. 
As another point of contrast, the specific problem presented to the experts in this study required only that they 
“translate” existing instruction to a new format, rather than “design” something new from scratch. Additional 
research is needed to help clarify these seemingly conflicting results. 
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Implications for the Education of Designers 

The results of this study have implications for the education of instructional designers. First, the results 
support findings from previous research (LeMaistre, 1998; Perez & Emery, 1995) that suggest that ID experts 
construct a clear understanding of the issues in an ill- structured problem as the first step in the problem-solving 
process. While two of our experts did not specifically articulate a synthesis statement, they all narrowed the problem 
space by identifying key problem elements in the case narrative. It is possible that, with practice and support, 
novices also can learn to do this effectively. This idea is supported by findings from Dufresne, Gerace, Hardiman, 
and Mestre (1992) who taught students to solve physics problems using a computer-based “Hierarchical Analysis 
Tool” (HAT) that prompted them to analyze the problem in more expert-like ways. In subsequent problem-solving 
exercises, students who received the HAT instruction were noted to use the expert-like strategy more often than 
students who received other types of instruction. That is, by using a hierarchical analysis structure that integrated 
concepts, principles, and procedures, novices were able to increase their focus on the deep structure of the problem 
rather than on surface details. A similar approach may be used with ID novices who could be given analysis 
guidelines that compel them to consider “big picture” (as opposed to surface) issues when analyzing ill-structured 
problems. Preliminary results from a study by Author (in progress) suggest that the use of guidelines or scaffolds 
that remind novices to, among other things, 1) focus on the “big picture,” 2) consider the core issues (i.e., those that 
are most central to understanding the case), and 3) consider the critical issues (i.e., those that are most likely to lead 
to a successful resolution), helped them analyze an ill- structured case scenario in more expert-like ways than novices 
who were not given these guidelines. Additional research is needed to verify these preliminary results. 

The experts in this study appeared to use personal frames of reference, based on accumulated sets of 
previous experiences, when conceptualizing the ID challenges embedded within a problem scenario. While it is 
unlikely that ID students will have amassed many personal experiences (related to ID practice) while still in school, 
there is some indication that students can benefit, vicariously, from the experiences of others (Jonassen, 1999; 

Schon, 1993). Fortunately, there are a variety of ways to incorporate both direct and vicarious learning experiences 
into our graduate programs including the use of case studies; internship and practicum experiences; guest speakers; 
as well as consulting with, and working for, real clients as part of a studio design approach. This is in line with 
Collins’ (1991) recommendations for designing cognitive apprenticeships, which suggest that novices observe 
experts as they solve problems so they can witness the false starts and dead-ends that are typical of real-world 
problem solving. Furthermore, by hearing experts’ reflections-in-action (Schon, 1993), novices may gain deeper 
understanding of the entire problem-solving process. So, for example, if students were to view videotapes of experts 
as they analyzed ill- structured case studies, they would not only see real-world examples of completed analyses, but 
also hear how the experts arrived at their final solutions. Furthermore, students could compare their ideas about the 
case with those of experts, providing rich fodder for meaningful reflection. Each of these strategies could provide 
opportunities for novices to hear and benefit from the experiences of more expert others. 

Still, it may be important to help students index these experiences in ways that are readily retrievable 
(Kolodner, Owensby, & Guzdial, 2004). According to Jonassen and Hemandez-Serrano (2002), the ability to recall 
prior experiences depends on how those experiences are stored in memory. Aamodt and Plaza (1994) noted that 
effective case-based reasoning requires a “well thought out set of methods” (p. 41) for indexing cases/experiences so 
they can be readily integrated into existing knowledge and then easily retrieved when needed to solve similar 
problems. Since only a subset of one’s knowledge and/or experience will be relevant to any single problem, a 
practitioner needs to be able to select and retrieve only that which is relevant. Part of the job of the ID educator, 
then, becomes that of supporting novices as they observe, accumulate, and store (in memory) relevant experiences 
during their graduate programs. 

Providing appropriate learning experiences may be part of the answer, but it is also important to help 
students reflect on those experiences in ways that enable them to readily recall and use those experiences during 
future problem solving situations. For example, case study discussions can focus students’ attention on specific 
problem elements and design challenges in the case, as well as possible solutions and their implications. Following 
this, Didierjean and Cauzinille-Marmeche (1998) recommended that we help students represent the knowledge 
gained at multiple levels of abstraction. At the lowest level, this would entail simply storing / indexing 
“unabstracted” knowledge in the form of specific case details (e.g., context, stakeholders, events). At the next level, 
case details would be represented in the form of themes or concepts underlying the specifics (e.g., communication 
issues, project management decisions). Further abstraction might result in the formation of a principle, or rule of 
thumb, that links the concepts or themes through causal, correlational, or chronological relationships (e.g., “If the 
subject is controversial, the delivery should be face-to-face; if it’s a run-of-the-mill topic, it could be converted to 
the Internet.”). As students’ case knowledge becomes more abstract, it would have the potential to be more 
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generalizable. However, it also would lose the local and specific nuances that would enable it to address novelties, 
ambiguities, and exceptions to the generalities (Didierjean & Cauzinille-Marmeche, 1998; Reimann & Schult, 

1996). Ideally, students would integrate their knowledge across all levels of abstraction to enable the most effective 
reasoning and recall. As students’ mental models became more sophisticated through each subsequent experience, 
they would, over time, increase their ability to identify the underlying structure of the problem situation and thus 
reason from first principles (Reigeluth, 1997). 

Limitations and Suggestions for Future Research 

In addition to the small number of participants, a primary limitation of this study relates to the use of think- 
aloud protocols for data collection. While this is a common approach used in expert-novice studies (LeMaistre, 

1998; Perez & Emery, 1995; Rowland, 1992), some participants are better at engaging in the concurrent tasks of 
problem solving and thinking aloud than others. In this study, this may have been complicated farther by the manner 
in which the design problem was presented to the participants, that is, as a text-based case narrative. Asking 
participants to think-aloud as they read the case out loud may have interfered with their normal problem solving 
processes. Furthermore, according to Lloyd, Lawson, and Sean (1995), protocol analysis itself may interfere with 
designing and thus not accurately represent the design thinking we are trying to analyze. Additional research is 
needed to determine the extent of the influence of a think-aloud procedure on design thinking, as well as to evaluate 
other means for capturing the thought processes of experts. For example, it may be possible to videotape a team of 
designers as they discuss a novel case, capturing their conversations, and thus their thinking, as they analyze the 
situation. Another option might be to present the case situation via video rather than print, perhaps enabling a 
different type of thought process than that captured by reading and thinking-aloud. Finally, asking individuals to 
write a synopsis of a case situation might be able to capture important components of the problem-solving process, 
such as initial design decisions or elements deemed critical by the designer. While each of these methods has 
advantages and limitations, it also may be possible to combine approaches to adequately address critical 
shortcomings. This area appears ripe for additional research. 

While this study focused primarily on experts’ approaches to analysis (problem- finding), it will be 
important also to look at experts’ approaches to designing solutions to the problems identified. Although all of the 
participants in this study made some suggestions about how to solve the problems described in the case, this was not 
pursued in depth due to time constraints. Future research should examine how experts use their previous knowledge 
and experiences to design solutions to ill- structured problems, including the extent to which they apply personal 
rules of thumb as they did during the problem finding process. 

Conclusion 

The results of this study suggest that experts tend to quickly filter through the layers of a problem situation 
to determine the key elements, by drawing on their previous knowledge and personal experiences, as a first step in 
conceptualizing the ill-defined issues in a case-based problem. Regardless of whether individual, multiple, or 
composites of previous experiences were recalled, these were used to create personal rules which were not gleaned 
from knowledge or experience alone, but from a combination of the two. Specifically, knowledge and experience 
appeared to facilitate the problem solving process by providing the individual with: 

1 . A personal perspective or frame of reference, incorporating a mental model of the ID process, that 
guided the individual’s thinking about the problem and 

2. A set of idiosyncratic rules of thumb that helped the individual reflect on specific ways to address the 
complex issues in the case. 

According to Dufresne et al. (1992), “It is the organization and use of knowledge, not the knowledge itself, 
that play the pivotal role in successful problem solving’ (p. 330). This suggests the need to rethink the way we 
scaffold student problem solving activities in order to more effectively enable them to organize their domain 
knowledge in ways that facilitate more expert problem solving. Based on the results of this study, we suggest three 
specific strategies for the education of designers: 1) helping students conceptualize the key issues in an ill-structured 
problem by scaffolding their analysis efforts to be more expert-like, 2) helping students accumulate a variety of ID 
experiences, either directly or vicariously, from which they can draw when faced with an unfamiliar design 
situation, and 3) enabling students to index these experiences in ways that facilitate efficient recall of relevant cases 
and/or principles when solving future ID problems. It is our hope that the use of these strategies will lead to more 
skillful problem- solvers who are able to strategically apply their knowledge, whether from textbooks or vicarious 
experiences, to articulate clear conceptualizations of ill-structured ID problems and, ultimately, to generate powerful 
and effective solutions. 
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Through Singaporean Eyes: How Young Adults Learnt Online 

Dr Linda Mei-Ling Fang 



Abstract 

As culture has been known to shape learning, it would invariably influence online learning as well. 

Learners bring along their culture to any learning environment. As they are influenced by multiple layers of culture, 
it is uncertain how these influence online learning. This qualitative descriptive case study investigates how a group 
of 20 young adult Singaporean Chinese in a Speakers’ Program offered as blended learning, were learning online. It 
examined the influence of national, ethnic and cyber cultures on their online learning experience. The findings 
suggest that the learners’ culture influenced how and what they wanted to learn, and with whom. While the 
influence of the different cultures predominated in certain areas, they often had a competitive or complementary 
effect on each other. 



Introduction 

The culture of the learner needs to be considered when explaining the individual’s learning (Schunk, 2004). 
This is because culture shapes the mind by providing the toolkit by which individuals’ worlds are constructed, as 
well as conceptions of themselves and their powers (Brunner, 1996). As cultural determinants such as gender, 
ethnicity, social class shape learners (Grenfell, 2004), they have been observed to learn differently (Brislin, Bochner 
& Lonner, 1995). 

Culture is known to influence preference for learning and training (Marquardt & Kearsley, 1999). As 
behaviour is predetermined by mental programs or the software of the mind, culture can determine how students 
interact with their teachers (Hofstede, 1994). For instance, power distance influences how the students regard their 
teachers. Uncertainty avoidance differences will determine the level of expertise a teacher is expected to have. 
Students from collectivist cultures are more reluctant to speak up in class and instead try to preserve harmony and 
maintain ‘face’. Masculinity and femininity also determine whether students present themselves as competitive or 
not. Hence, recognition of learner’s personal epistemologies and conceptions of learning is important because 
learners’ beliefs, significantly influence the approach they take to learning as well as what they ultimately learn 
(Pratt, 1998). 

Culture influences online learning as well. For instance, Lim, (2004) found differences in online learning 
motivation between American and Korean learners. The American learners indicated significantly higher motivation 
scores for course relevancy, course interest, reinforcement and self-efficacy compared to the Koreans, who in turn 
scored significantly higher for learner control. For affect, the American students comparatively felt a greater sense 
of accomplishment when completing online lessons, preferred voicing personal opinions during class, enjoyed 
learning and enrolling in classes to obtain a sense of belonging. This shows that the Asian e-leamer has a distinctive 
approach to learning online. Edmundson (2004), who studied the impact of cross-cultural dimensions in online 
learning, found that learners from two opposing cultures, namely USA and India, were able to achieve equitable 
learning outomes after taking a Western-designed online learning course. However, national cross-cultural 
differences were found to have affected learners’ preferences and perceptions in educational contexts. Other studies 
have focused on the cultural aspects of user interface and understanding of a website involving international user 
groups (Evers, 2002), the relationship between national culture and usability in online learning (Adeoye, 2004), and 
differences between American and Chinese university students’ design of personal web pages (Chu, 1999). Most 
studies used the quantitative approach. More qualitative studies like Chu’s (1999) are needed to provide a balanced 
picture of online learner. 

In Singapore, there is widespread use of technology in everyday life (Corbett & Wong, 1999) and in the 
schools (Hung, Tan, & Chen, 2003). Singaporean young adult learners belong to a cyber generation that has grown 
up with computers. As individuals are influence by different layers of culture (Hofstede, 1994), it can be expected 
that students in highly-networked, multi-cultural Singapore could be influenced by the national, ethnic and cyber 
cultures at the same time. While there have been studies on how Singaporean students interact online compared to 
their US counterparts (Tan, Wei, Watson, Clapper, & McLean, 1998), there are no studies that consider how online 
learning is influenced by three levels of culture. 
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This qualitative, descriptive case study focused on 20 young adult Singaporean Chinese enrolled in a two-month 
long Speakers’ Program offered as blended learning. The research question was as follows: How were young adult 
Singaporean Chinese at a polytechnic in Singapore learning online? This study was part of a larger study which 
focused on the online learning experience of these learners (Fang, 2006). It surmised that the learners’ culture 
would influence how they were learning. Hence, the conceptual framework was as follows (see Figure 1): 




Figure 1. Conceptual Framework 

Terms Used 

i) Singaporean Chinese: Those who have been bom in Singapore, lived in Singapore and educated in the national 
schools are referred to as Singaporean Chinese in this study. 

ii) Student Ambassadors Program: A specially-designed speakers’ program to prepare student for public speaking 
events with the aim of promoting their institution, school or diploma; upon successful completion of the 
program, the Student Ambassadors would undertake a range of speaking assignments for both formal and 
informal events, addressing small groups of two to three persons to large audiences of 1000. They would have 
to be prepared for the unique demands of each event. 

iii) Senior Student Ambassadors: Students who have successfully completed the program and have served well as 
Student Ambassadors. 



Literature Review 

How learning takes place online will be briefly reviewed. Culture at the national, ethnic and cyber levels 
would be reviewed. 

Online learning 

Online learning calls for an active approach to learning (Brooks, Nolan, & Gallagher, 2001; Dringus, 

2000). It occurs where there is interaction with non-humans (i.e., content) as well as humans, namely tutors and 
other students (Garrison & Anderson, 2002; Moore, 1989). Each type of interaction has its role in ensuring that 
learning takes place. Moore (1989) explains that interaction with materials results in the change in learners’ 
understanding, perspective, or the cognitive stmctures of the learners’ mind. Interactions with the instmctor (the 
expert) allows for the presentation of information, demonstration of skill or modeling of certain attitudes and values, 
and the shaping of learning. In huge classes, interactions with other learners at the group level can lead to learning. 

Online learning applications could range from simple presentations to sophisticated online learning 
simulations with graphics, animation, video, and audio components (Hall, 1997). Interaction with others could take 
place in formal or informal learning communities via e-mail, list-serves, threaded discussion forums (Joliffe, Ritter, 
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& Stevens, 2001), live online chat sessions and discussions on whiteboard (Long, 2004; Welsh, Wanberg, Brown, & 
Simmerin, 2003) and video-audio conferencing (Duggleby, 2000). The interaction with other learners allows for 
collaborative work and discussions to take place online (Dillenbourg, 1999). Attention has to be paid to developing 
the sense of community within the group for the learning process to be successful (Pallof & Pratt, 1999). Learners 
receive feedback from humans or through electronically programmed means (Long, 2004) in a variety of ways: 
email messages, conference and announcement postings, automated feedback in the form of schedules, news, 
updates and personal feedback (Dringus, 2000). Guidance can be provided through e-moderation (Salmon, 2000), e- 
coaching (Bagshaw & Bagshaw, 2001; Dennen & Wang, 2002) and /or e-mentoring (Nuernberger, 2000). 

Existing learning theories can be easily applied to computer-mediated technologies as the learning 
environment for drill and practice (Behaviorism), information transmission and processing (Cognitivism), personal 
discovery of knowledge (Constructivism), and the social construction of knowledge through mediated by different 
perspectives (Social constructivism) can be provided (Chen & Hung, 2003). However, Social Learning (Bandura, 
1972) or Social Cognitive Theory (Bandura, 1985) which involves observational learning has yet to be explored 
fully. 

Culture 

Singaporeans are generally collectivist and high in power distance (Hofstede, 1994). They desire success 
in their studies and view education as a means to an end i.e. achievement in education would help them to get ahead 
in life (Kau, Jung, Tambyah, & Tan, 2004). 

The Chinese value education (Nisbett, 2003). Teachers, who are highly respected, are expected to teach as 
well as guide students. They are the deciding authority and power on what knowledge to teach. Students readily 
accept the information and rarely question or challenge teachers in the classroom (Chan, 1999). As Chinese culture 
values modesty, Chinese students in the past would regard themselves “not worthy before their teachers”. This 
modesty also causes students not to express their true opinions so as not to embarrass or offend others. In addition, 
classroom behavior serves to maintain face and prevent shame. For instance, students “lose face” for misconduct 
and poor performance (Chan, 1999). That explains their quiet behaviour in class (Biggs, 1996). 

The cyber age offers conveniences in accessing materials. Digital technology allows broadcasts to be non- 
real-time, allowing for on-demand information to predominate. Electronic libraries, entertainment videos are all be 
available any time (Negroponte, 1995). Learners who are comfortable with the latest Internet technology and cyber 
culture, having embraced the telemetric world, can easily adjust to the 24 hour, multi-dimensional, global view of 
the world and are capable of learning when they want to, where they want to, and at their own time and pace 
(Harking, Turner, & Dawn, 2001). In addition, a new form of cyber communication with its emoticons and 
netiquette is emerging (Tapscott, 1998) and there is a new form of electronic literacy (Kiesler, 1997). Online 
communication influences the development of adolescents and new friendships are built online; cyber families and 
friends exist (Tapscott, 1998). 

Chinese Online Learners 

Culture influences the manner in which Chinese students interact online and they are more comfortable 
communicating online in a smaller group as there is more trust (Tu, 2005; Tu & Mclsaac, 2002). This “inner circle” 
in the Chinese culture provides care and protection (Bond, 1991). In cross-cultural studies, Chinese learners have 
been known to be less critical than their American counterparts and avoid confrontation (Thompson & Ku, 2005). 
Chinese heritage students at an Australian university were found to have posted more messages associated with 
organizational matters while their Australian counterparts posted more messages associated with the intellectual 
contributions of the discussion (Smith, Coldwell, Smith, & Murphy, 2005). 

Singaporean Online Learners 

Online traits of Singaporean learners have been noticed in studies. For instance, trainee teachers were 
observed to avoid participating and interacting actively online; they did not keep the discussion thread for prolonged 
asynchronous discussion and posted independent statements of content unrelated to the prior messages (Myint & 
Yeap, 2004). They tended to be task-orientated, having little or no interest in socializing through electronic means. 
While they had contributed independent ideas online, there was a lack of heated discussions or debates where there 
would be sides taken on issues, negotiations or compromise (Hew & Cheung, 2003). Singaporeans have also been 
found lacking in displaying critical thought online (Myint & Yeap, 2004; Soo, 2003). National culture has 
influenced how Singaporeans students interact online: they were more collectivist compared to their US 
counterparts; they were less willing to challenge the majority position; and the majority influence remained 
unchanged when CMC replaced verbal and visual communication (Tan et al., 1998). 
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Methodology 



The qualitative descriptive, single case study involved the 20 participants enrolled in a Speakers’ Program. 
They were first and second year Engineering students in a polytechnic in Singapore. All were Singaporean Chinese 
between the ages of 17 and 19, and had been educated in the local school system. The training program, which was 
designed to prepare them to be Student Ambassadors of the polytechnic, was offered as blended learning during the 
polytechnic semester break between September to October 2005. It comprised six weekly face-to-face sessions with 
accompanying online sessions, ending in a project presentation. They were taught face-to-face and tutored online by 
a trainer, who was the researcher and a member of the faculty, as well as senior Student Ambassadors. Materials 
were placed on Blackboard 6. The program was designed to include Behaviorist, Cognitivist, Individual and Social 
Constructivist learning theories. Hence, the participants were provided with a variety of learning approaches, and a 
range of opportunities to interact with the content, each other, the trainer and senior Student Ambassadors. 

Data was derived from their discussion board submissions, home pages, email messages, online chat, their 
personal records, and two interviews. The first interview was with individuals at the beginning of the program. The 
second interview was with the project group held at the end of the program. These were taped and transcribed. All 
data was saved on word documents. The individual interview, personal introductions, home page and personal 
records provided a profile of the participants. Data from their online responses provided evidence of their 
interactions with the content, each other and the training team. Data from a second interview revealed how they 
were learning, with whom and why. Data from these three sources were then triangulated. 

Findings 

The participants valued the program as it provided the desired skills would contribute to their personal 
development at work and in their studies. They were aware that the online work complemented the classroom 
sessions, having sensed the interconnectedness between these two mediums of learning. Hence, they were highly- 
motivated and conscientious in their assigned tasks. Responses were received during the day as well as late into the 
night, often after returning from their part-time job shift. They clearly determined what was leamt, from whom the 
learnt from or preferred to learn from, and how. These are reported under learning from the online content, each 
other and the SA Training Team, comprising the SA Trainer, the Online Coaches and Online Chat Leader. 

Learning From Content 

The participants were able to interact independently with the online materials, and attempted the required 
tasks and activities. Their submissions on the discussion board were excellent in quality, reflecting a maturity of 
thought. This showed that they had grasped the essence of the materials and were able to complete the exercises 
well. It was no wonder that they mentioned that they leamt much from the online materials and activities. The 
following exemplify their ability to leam quickly: 

i) They used the materials to think for themselves. They showed an ability to identify positive information about 
the institution they were in, and achievements of competing institutions. In addition, they were able to highlight 
the key reasons for joining their institution, through selection, ranking and elaboration. Their responses revealed 
much sensitivity and thoughtful reflection on the topic. Asked which key words and phrases could be used, 
they highlighted the “conducive learning environment”, “friendly lecturers”, “wide variety of diplomas to cater 
to all students needs”, “various interesting CCA [co-curricular activities] bring out [sic] the hidden talent in 
students”, “healthier lifestyle” and how it “prepares one for the ever-changing society”. 

ii) They leamt by exploring, searching and synthesising information on their own. A web-quest activity required 
participants to recommend what would be the best course in Singapore for Ling Ling, a fictitious character, to 
enrol in. The advice provided was exceedingly sound, convincing and mature. One participant, WS, wrote with 
much confidence: 

I would strongly suggest . . . [institution’s name and diploma course]. Since you are open to any engineering 
course, I assume you do have a clear sense on which field of engineering you want . . . [name of course] is a 
3yr full-time engineering course. The Course equips you with systems thinking as well as facilitation skills, 
both of which are essential in today’s market environment. Upon graduating from the course, you will 
possess the unique ability to integrate business and engineering principles to serve on cross-functional 
teams that drive improvements and productivity towards organisational excellence. Taking this course 
gives you more options because after completing the course, you will be able to know exactly what you 
want and to take which particular degree in Singapore’s University as you have experiences in both fields. 
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The course will definitely gives you an edge over others as you have both business and engineering 

knowledge. 

He went on to suggest several degree courses she could pursue at the two local universities upon completing the 
recommended diploma, highlighting the relevance of each to the working world. After mentioning the high 
ranking of the two universities, he concluded “so you can aspect [sic] quality education in Singapore. Hope I’ve 
give [sic] you some valuable information LingLing and Good Luck!” 

iii) They learnt quickly from patterns that were provided. In their “Speech Class”, they were quick to grasp the key 
principles of pronunciation that influence the intelligibility of Singaporean speech even though they had not 
received any formal training in pronunciation or linguistics. Yet, from the samples provided in one brief lesson, 
they could recognise the inherent problems with local pronunciation which could lead to miscommunication 
internationally. The participants responded with Singaporean examples, for instance, word pairs like “tin-thin”, 
“library-library” and “think-thing” that could influence the intelligibility of Singaporean speech. 

iv) They learnt from models provided. An example was when the participants selected their plan for their personal 
stories. Despite being presented with a variety of approaches for the introduction, the development of the speech 
and conclusion, all but one of the participants followed the same model in their first draft, submitting close to 
“cookie cutter” versions of their personal story as they were based on the same theme and event. For many, it 
was the first attempt at they were writing a speech, and they mentioned the sense of achievement they felt on 
their being able to write it. The model seemed to help them draft it quickly. 

v) They learnt from an awareness raising activity. One activity required the participants to make identify which 
story reflected what was most appropriate to say in public. They made sensible decisions as to the best types of 
stories. One participant declared that it truly sensitized her to what could be said in public. 

Overall, they seemed comfortable interacting with the content and could learn in a variety of ways. They could be 
considered as able to leam independently from the content. 

Learning From Other Participants 

Judging from the online submissions, the participants did not seem to have learnt much from each other. 

The lack of interaction may have limited learning directly from each other; however, it did not prevent them from 
learning indirectly from each other. 

Group size seemed to affect interaction. Online, they were guarded and cautious to interact in a large group. 
Being generally unfamiliar with the other participants, they were also reluctant to comment on each other as 
individuals or their work for fear of offending the other. For instance, “Positive Vibes”, an activity that provided the 
opportunity for public affirmation and appreciation of the other caused much discomfort amongst the participants. 
Neither did they engage in any issues raised. This absence of interaction prevented participants from contributing to 
the learning from each other. However, learning from each other was largely indirect, and hence vicarious. The 
participants learnt by reading each other’s submissions, and from the comments and feedback given to others by SA 
Trainer and SA Online Coaches on the discussion board. Some read what their friends wrote out of curiosity, while 
others read the submissions to leam. Firstly, they leamt about each other without addressing them directly, and 
without their knowledge from their introductions, home pages and the organized chat. Secondly, the participants 
sometimes learnt by following the submissions of others. That was why the responses followed in quick succession 
after first submission. They looked for good answers, used them as models and followed the style or type of answer. 
At times, the first submission by a participant was taken as the model. For instance, many revealed that for the 
“WOW” factors activity, subsequent submissions were markedly similar in content and coverage to the first 
submission. No one tried to be different or leave out any information. It was probably a “safety in numbers” 
reaction. Also, when in doubt, the strategy was to look for the best submissions to follow. One pair confessed 
selecting the best answers, checking them, summarizing them and resubmitting the information as theirs. 

They seemed to have leamt from each other in smaller groups. However, while they reported sharing ideas 
and working together in project groups for their final presentation, there was no evidence of “work in progress”, 
discussions reflecting how they worked on their project on the discussion board. Whatever was submitted as 
responses which were observable to all, were found to be in their final, summative stage. Formative work that 
emerged from the shaping and exchanging ideas was not visible because these possibly took place in private, within 
the confines of their project group. Perhaps that was where they could freely discuss, agree or disagree. For 
example, two participants mentioned sending each other ideas and thoughts through email for their group 
assignments, but they also communicate through SMS messages and over the phone. Others discussed privately in 
their project groups via MSN messenger. Because the data was not available, it had to be assumed, such learning 
could have taken place, judging from the quality of their group presentations. 
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Learning From the SA Training Team 

Different types of learning were derived from the members of the training team, comprising the Trainer (a 
staff member), and Online Coaches and Online Chat Leaders, who were senior Student Ambassadors. The context 
was an influencing factor. 

The Trainer seemed to play a more consistent and dominate role in helping the participants learn in the 
formal setting in a large group. She consistently provided feedback or comments each time a participant posted a 
submission on the discussion board. However, the participants appeared to be passive learners, merely accepted the 
comments, feedback and suggestions, occasionally asking for clarification. They never challenged what was said. 
None of them pursued any matter or were engaged in a discussion with the Trainer. She had little influence in an 
informal learning situation. Apart from the occasional email requesting clarification, the Trainer was not invited to 
be on MSN with the participants when they chose to learn in small groups. Hence, the Trainer was not able to shape 
their learning or help them for discussions of work in progress in informal settings. 

In addition, the participants showed a remarkable ability to learn quickly from the Trainer’s feedback. This 
was evident when they scripted their personal story. Each participant was given individual feedback on the first draft 
personally via email. All revised versions showed marked improvement. The most evident change was in the 
number of words. Chart 1 below shows the word length percentage change in relation to their initial drafts. The 
participants have been ranked from 1 to 19 ranging from the greatest number of words reduced to the greatest 
number of words increased. One participant, who was ill, submitted her work after the program had ended and thus 
was not included in the data. 




Chart 1. Change in Story Length 

Note: Chart shows submissions of 19 participants 



Once the participants learnt to include or exclude, their speeches became more focused. They then were then able to 
move away from the model used and borrowed sentences to incorporate more independent thought and personal 
experience. They sounded more convincing and confident in their revised speeches. One participant who did not 
follow any model and struggled to make sense of the writing process had a credible personal story. His revised draft 
showed much promise as it had the most original ideas and interesting insights. Hence, individual feedback allowed 
the participants to produce interesting, differentiated personal stories in a very short time. 

The Online Coaches seemed to play a much less influential role, particularly in public, on the discussion 
board, rather than in private, on MSN. Their task was to provide assistance when needed. The participants hardly 
responded to the offers for help from the Online Coaches in public, however, they had some success in helping the 
participants in private. One member of an all female project group mentioned receiving encouraging comments and 
feedback on their drafts via email and passed them on to the others via a variety of sources, namely, email, SMS and 
phone. Another group benefited from the useful tips sent by their Online Coach though they did not engage him in 
their discussions. Another group chose not to bother their Online Coach. Two groups who chatted with their Online 
Coach on MSN reported that their assigned Online Coach to be very helpful. It was assumed that these groups 
received more help. The extent of the assistance could not be fully determined since the data of their invisible 
interaction was not available to the researcher. 

Participants learnt least from their Online Chat Leader because they viewed the Organised Online Chat as a 
means to socialise, although it was based on a rather serious topic. Placed in groups of 1 1, they were chatty, and at 
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times unfocused during their discussion. Some claimed that their SA Online Chat Leader was very knowledgeable 
and had provided them with information on the topic for discussion. However, this was because their feigned 
ignorance was merely a ploy to increase social interaction with him; hence, it might not have increased their 
knowledge significantly. 

Online Learning Mode 

Seemingly passive learners online can be in actual fact, are active learners. If learning is only associated to 
visible interaction, not all the learning will be captured. The “invisible” learning could provide insights as to the 
participants preferred to learn. Hence, their online learning mode, depicted in Table 1, attempts to describe how the 
participants were learning, by reflecting the direct and indirect learning that took place online. 

Table 1 

Preferred Mode of Learning 



Source of Learning 


Direct 


Indirect 


Comments 


Content 


Predominantly from 
materials and activities 


NA 


Learnt directly and 
effectively in a variety of 
ways 


Other participants 


Limited in large group 
More with project group 


Learnt by reading online 
submissions of other 
participants 


Learnt directly from 
members in small group; 
learnt indirectly from how 
others responded in a large 
group 






Learnt by reading 
comments SA Trainer and 
Online Coaches made to 
these submissions 


Learnt indirectly from 
feedback provided to other 
participants’ responses 


SA Trainer 


From feedback of online 
submissions 




Learnt directly in formal 
context 


SA Online Coach 


Limited, from comments, 
suggestions 

Able to work with project 
group, if included in 
discussions 




Learnt directly 
Less in formal situation; 
possibly more in an 
informal situation if 
familiar with or invited by 
group members 


Online Chat Leader 


Not at all 


Not at all 


Chatting not considered a 
learning situation 



Sensing Participants’ Preferences 

The participants’ preferences for what they wanted to learn, how and with whom, influenced the way they 
reacted to the materials, tasks and activities. This resulted in the outcomes differing from the intended design of the 
lessons. There were times when the activities had a life of their own. For example, their introductions, which were 
on uploaded Microsoft word documents on the discussion board as well as on their home page, were designed to 
provide an opportunity to present themselves positively in public. Instead, they used the opportunity to reach out to 
others and included personal details and photos. In addition, the organised online chat, which was expected to be a 
rather serious discussion, turned out to be an enjoyable, interactive activity and out-of-focus discussion as the 
participants took it lightly. On the other hand, there were some activities that made them uncomfortable, and the lack 
of response signalled this. For instance, they either avoided invitations to comment on the participants or their work 
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or circumvented the issue by providing generalised statements. When asked to identify the worst story provided in 
the materials, they deliberately avoided commenting, claiming that they found none at the point of submission. 
Neither did they discuss their work online in a large group, choosing to do so in private. At times, they were held 
back by their shyness. The few bold attempts to reach out to the others in their introductions were not reciprocated 
and hence no friendships were formed online. 

Discussion 

The participants learnt through online interaction with non-humans (i.e., content) as well as humans, 
namely tutors and other students (Garrison & Anderson, 2002; Moore, 1989). Firstly, interaction with materials 
resulted in the change in learners’ understanding, perspective, or the cognitive structures of the learners’ mind 
(Moore, 1989). They seemed an effective mode of learning for the participants who could learn from a variety of 
resources provided and activities. Secondly, interactions with the instructor (the expert) allowed for the shaping of 
learning (Moore, 1989). However, this happened only in a formal learning context. They were not able to learn from 
their Online Chat Leader but managed to learn from their SA Online Coaches in a limited way, more with increased 
familiarity. Thirdly, while interactions with other learners at the group level was expected to lead to learning 
(Moore, 1989), the limited interactions resulted in learning to be direct in a small group and indirect in a large group. 
Overall, there was an active approach to learning (Brooks et al., 2001; Dringus, 2000) for learning that was public 
and visible (on discussion board) or private and not visible. Despite the design of the program based on different 
learning theories (Chen & Hung, 2003), the learners still influenced what and how they were to learn, and from 
whom. For instance, social constructivism possibly took place online within the smaller groups, while social 
learning and social cognitive learning (Bandura, 1976, 1985) or vicarious learning seemed to be reported in large 
groups. Perhaps this was why one participant aptly described learning online as a “visual” experience. 

Cultural Interpretations 

Culture could have subtly shaped what the participants, how and with whom. National culture was evident 
when the participants seemed to respond in the same manner, their inclination to collectivism (Hofstede, 1994). 
Chinese culture also predominated in several situations. Their respect for the SA Trainer as an authority (Chan, 
1999) was reflected in the way they followed a model she provided for their speech writing and incorporated her 
feedback in their revision. The participants were quiet in a large class (Biggs, 1996), refraining from expressing their 
true opinions to avoid embarrassing or offending others, or even “losing face” for themselves (Chan, 1999). Their 
preference for learning was in their smaller group (Tu, 2005; Tu & Mclsaac, 2002). Cyber culture was also evident. 
Learning could have taken place at any time (Harking, Turner, & Dawn, 2001) judging from the responses that were 
received in the day and night, even in the early hours of the morning. In addition, their casual communication 
included the use of emoticons in their personal introductions and online chat online (Tapscott, 1998) probably 
shaped their view that chatting was appropriate for social activities and bonding but not to engage in serious 
discussions for learning. 

The influence of multiple levels of culture could have a complementary or competing effect. Similar 
cultural values at different levels could have reinforced what they learnt, how and with whom. For instance, the 
desire for success in studies (Kau et al., 2004) and the respect for learning (Nisbitt, 2003) could be the reason the 
participants took content materials and tasks seriously. In addition, high power distance (Hofstede, 1994) and 
respect for authority (Chan, 1999) resulted in their desire to learn from the SA Trainer only in a formal learning 
situation. At other times, they could have played a competing role. For instance, the opportunity for online 
friendships (Tapscott, 1998) were not realised in a large group, as the need to belong to a smaller group (Tu, 2005; 
Tu & Mclsaac, 2002) seemed stronger. That was the most likely reason for their tendency to leam vicariously in a 
large group and directly from each in a smaller group. 

Conclusion 

The findings suggest that culture at the national, ethnic and cyber levels could have influenced the 
participants online in terms of whom or what they were learning from, and the preferred mode of learning. The 
different levels of culture could have influenced the participants directly or indirectly, consciously or unconsciously 
and could have a reinforcing or competing effect on each other. 
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Abstract 

National testing efforts indicate that students are lagging in developing adequate writing skills. Research that has 
been conducted on the use of word processors to improve writing skills at different grade levels has yielded mixed 
results at best. The purpose of this study was to determine the effectiveness of an online essay scorer in improving 
the writing of high school sophomores. Regression results found the automated online essay scorer combined with 
Group membership (experimental or control) explained 40% of variance, /?<.01. 

Introduction 

According to the 2002 National Writing Report card, two-thirds of all children in the United States perform 
below the proficient level in writing, a level that indicates solid academic performance. Of all students, 75% perform 
at the Basic level, which indicates minimal effectiveness in getting main ideas across. These results do not differ 
significantly from the 1988 National Report Card (Applebee, Langer, Jenkins, Mullis, & Foertsch, 1990) wherein 
students at grades 4, 8, and 12 demonstrated poor writing abilities. Less than 45% of students across the nation 
wrote responses to informative prompts that were considered adequate or better. Additionally, only 27 to 36% of 
students provided responses to persuasive prompts that were considered adequate or better. Although students 
performed better on imaginative narrative responses, merely 9 to 56% performed at a level considered adequate or 
better. The low performance of students on the National Writing Report card across a time span of 12 years indicates 
that educators need to examine methods to improve the writing skills of our nation’s learners. 

Composition research initially experienced growth in the 1960s when researchers began to study the 
processes in which professional writers engage while producing written text (Humes, 1983; Tchudi & Mitchell, 
1989). From these observations several different researchers developed writing as a process using both qualitative 
(Emig, 1971) and quantitative (Hillocks, 1984) research methods. This description of writing as a cognitive process 
in which writers go through a series of stages (i.e., prewriting, drafting, revising, and publishing) has dominated 
writing research for the past 46 years although it has been presented in various formats with emphasis on different 
components. 

Instructional methods such as sentence combining (O’Hare, 1973) and scaffolding (Langer & Applebee, 
1986) have been promoted, but these methods have been incorporated to varying degrees into the writing process 
approach, and so no viable alternatives to the writing process have been promoted at the secondary level. Research 
to date has emphasized that writing is a complex process that can be studied with many different methods, 
qualitative (e.g., Emig, 1971) and quantitative (e.g., Hillocks, 1984). The cognitive processes engaged in by writers 
are recursive and guided by goal setting and multifarious influences such as the task environment and long-term 
memory (e.g., Flower & Hayes, 1986). Furthermore, educators, as well as researchers, must acknowledge and 
accommodate this multifaceted progression if it is to be taught and studied effectively. Research has additionally 
indicated that writers can be aided by clear, concise, learner-centered goals and activities (e.g., Langer & Applebee, 
1986; Hillocks, 1984). Student writers seem to benefit from the consistent application of scales and scaffolds (e.g., 
Hillocks, 1984: Langer & Applebee, 1986) as well as teacher commentary that suggests techniques for improvement 
but gives control over the writing decisions to the students (e.g., Straub, 1997). 

Computer-assisted Writing 

The mid-1980s through the present have produced many studies that examined the effects of incorporating 
new technology, namely word processors, into composition instruction (Dybdahl, Shaw, & Blahous, 1997). 
Naturally, the advent of word processors and the wider availability of computers in classrooms encouraged 
educators to incorporate these tools into the composition classroom. 
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Computer-assisted instruction is a method that has yielded mixed results with regard to writing instruction. 
Numerous quantitative and qualitative studies with various results have been conducted on the merits of computer- 
assisted writing instruction when used to aid students in developing writing skills (Grejda & Hannafm, 1992; 

Owston & Wideman, 1997; Dybdhal, Shaw, & Blahous, 1997). Although the use of word-processing has 
encouraged students to make both mechanical and organizational revisions throughout text, “word-processing may 
inadvertently direct proportionately more attention to structural than holistic aspects of writing” (Grejda & 

Hannafm, 1992, p. 148). More sophisticated forms of composition software may help alleviate this issue by 
providing feedback on issues of coherence or unity through comments on organization, thesis sentences, and word 
choice. 

Frequent usage of word processors has, in fact, been shown to improve both the surface features of writing 
including grammatical structures, spelling, usage, and mechanics as well as the deeper structure which include 
general writing development, sense of audience, purpose for writing, overall meaning, unity, detail, and cohesion 
(Owston & Wideman, 1997; Lowther, Ross, & Morrison, 2003). Among the benefits of using word processors in 
composition classrooms are the relief of technical concerns for students such as handwriting and spelling (Gupta, 
1998; Hansman & Wilson, 1998). In addition, the ability to revise text without having to entirely rewrite the product 
encourages frequent revision and modification of written work (Owston & Wideman, 1997). 

Online Essay Scorers 

Recent advances in technology have given rise to online essay scorers such as the Educational Testing 
Service’s (ETS) Criterion as well as SAGrader. These scorers claim to give holistic scores, scores that include 
comments about content as well as about style, usage, mechanics, and organization and development. Students 
receive their feedback in about 20 seconds and are permitted to resubmit work for higher scores. Little research (c.f., 
Attali, 2004; James, 2006) currently exists that examines this new technology. Sophisticated online essay scorers 
have the potential to provide many of the conditions that previous research promotes in composition instruction. The 
feedback provided by the online scorer is nonjudgmental and is designed to provide support to students in making 
revision choices, not dictate revision choices. This is in line with research findings about effective commentary on 
student writing (Straub, 1997). Teachers may make comments on student writing, but the online grader assigns the 
score according to consistent scales. Consistent usage of scales may lead to internalization of the scales by the writer 
(Hillocks, 1984: Langer & Applebee, 1986). The scorer provides concrete examples of different elements it scores, 
such as topic sentences and transitional devices. These elements provide scaffolds to writers that lead them to higher 
levels of development while giving students the freedom to choose whether to access them or not (Straub, 1997). 

A statistical analysis of Criterion , ETS’s online scorer, in which 9000 essays were analyzed to assess the 
effectiveness of the feedback provided by the essay scorer, indicates that students are responsive to feedback for 
errors and are able to correct identified errors in successive drafts (Attali, 2004). Specifically, students were able to 
decrease grammatical errors, increase main points and supporting ideas, and also increase essay length. 



Purpose of the study 

The purpose of this study was to examine the use of a relatively new resource for evaluating writing, an 
online essay scorer. These scorers are more sophisticated than their predecessors and claim to offer holistic scores 
that include a basic analysis of grammar and spelling. Online essay scorers are an expensive resource for schools 
and their implementation in secondary settings is relatively unstudied. One primary research question guided this 
study: Is there a significant difference in experimental students' writing ability as assessed by human and online 
scorers when compared to that of control group students? 

Method 



Participants 

The experimental participants were 63 sophomore English students who ranged in age from 14-16. Male 
and female students were included. African-American and Caucasian students made up the majority of the students; 
however, Hispanic, Asian, and Native American students were also included. The majority of the students had no 
significant health concerns. A small minority of the students were inclusion students. 
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The control group participants attended a mid-southern urban Title I school that is primarily African- 
American in population. The participants in the experimental group attended a mid-southern urban Title I school that 
has an ethnically mixed population. Approximately two% of the students are Asian, Native American or Hispanic, 
of the remaining students about half are African American and half are Caucasian. 

Instrumentation 

Two essays were collected from participants in the experimental group and control group - an initial 
estimate (pre-assessment) and a follow-up estimate (post assessment). All essays were submitted for two methods of 
scoring. First, essays were submitted to an online essay scorer. Secondly, essays were reviewed by an Advanced 
Placement English Composition reader using the Tennessee Comprehensive Assessment Program (TCAP) Writing 
Assessment rubric. 

Online Essay Scorer. The online essay scorer was Criterion Online Writing Evaluation provided by ETS. 
Criterion uses automated scoring technologies to provide immediate constructive feedback online using a scale of 1 
to 6. A score of 1 is low and a score of 6 is high. Students receive annotated diagnostic feedback on grammar, usage, 
and mechanics. Criterion provides a reference guide that contains basic grammar rules, usage, and mechanics. 
Students may receive instructor feedback online in conjunction with comments made by Criterion. Students are able 
to correspond with instructors online. 

Teachers can choose the types of feedback that students receive and control the number of submissions 
students may make. Teachers also have the option to set the time period during which submissions can be made. 
Teachers receive a class error report for each writing assignment. Specifically, Criterion provides feedback on the 
following areas: 

a. Grammar, specifically sentence fragments, garbled sentences, subject- verb agreement, verb-form errors, 
possessive errors, wrong or missing words; 

b. Word usage including article errors, confused words, wrong form of word, faulty composition, nonstandard 
verb or word; 

c. Mechanics, including spelling, missing capitalization of proper nouns, missing initial capital letter in a 
sentence, missing question mark, missing final punctuation, missing apostrophe, missing comma, missing 
hyphen, improperly fused words, improper compound words, duplicate words; 

d. Style, including repetition of words, inappropriate words or phrases, passive voice, long sentences, short 
sentences, sentences beginning with coordinating conjunctions; 

e. And organization and development such as introductory material, thesis statement, main ideas, supporting 
ideas, conclusion, transitional words or phrases. 

Two scores from one to six for each participant were recorded — one for the initial estimate and one for the 
follow-up estimate. 

Advanced Placement Scorer. The scoring rubric used by the independent scorer was developed by the state 
of Tennessee to score the TCAP writing assessment. It also uses a scale from one to six; one is Deficient and six is 
Outstanding. The scorer was provided anchor papers, sample essays for each score from one to six as a reference for 
the scorer, used by the state. 

Writing Prompts. The prompts were selected from past prompts used by the state on the 1 1th grade TCAP 
writing assessment. Prompts were selected by the researchers for two purposes. One, older prompts were selected to 
limit the possibility that students had recently written about them on the state writing assessment. Two, the topics, 
banning cell phones at school and the possible elimination of extra curricular sports were selected because of the 
relevance these two topics have to students’ personal lives since the essays were to be written in the persuasive 
mode. The prompts were used at both the experimental and control schools. A copy of the prompts is available in 
the appendix. 

Procedures 

A writing prompt on whether cell phones should be banned from high schools was given to the control and 
experimental groups in October 2005 to gauge an initial estimate of writing ability. All essays were scored by 
Criterion and by the Advanced Placement Scorer. The control group proceeded with its normal curriculum. The 
experimental group continued to use Criterion when appropriate for the next 5 months. All writing samples were 
collected from students after they had been scored by Criterion. In March 2006, a writing prompt on the possible 
elimination of extra curricular sports was given to the control and experimental groups to gauge a follow up estimate 
of writing ability. The teacher of the experimental group taught some computer skills as needed, such as formatting 



98 




and other skills necessary to allow students to access Criterion both from school and other settings such as at home 
or the local public library. As part of the writing curriculum at the experimental school, students completed a 
minimum of one writing-across-the-curriculum lesson in each subject area each nine weeks that was submitted to 
Criterion. 

Analysis 



Three statistical tests were conducted on the data: correlations, regressions, and t-tests. The results will be 
organized in this order. 

The variables included in the data analysis were: 

a. two estimates of writing ability: (1) the initial estimate of writing ability and (2) the follow-up estimate of 
writing ability 

b. the two methods of instruction: (1) standard classroom instruction and (2) standard classroom instruction 
incorporating Criterion online essay scorer 

c. the two methods of scoring the initial estimate and the follow-up estimate of writing ability: (1) the human 
scorer and (2) the online automated essay scorer, Criterion. 

Results 

Descriptive statistics are provided for each of the data in Table 1 . The initial step in the data analysis was 
the running of correlations to look for a significant relationship among the variables (see Table 2). The three 
strongest relationships were: (a) the relationship between the initial estimate with the online essay scorer and the 
follow up estimate with the online essay scorer, reaching significance at r(61)=.449,/?<.001; the relationship 
between Group and the initial estimate with the online essay scorer, reaching significance at r(61)=.357,/?=.004; 
and (c) the relationship between Group and the follow up estimate with the automated online essay scorer, reaching 
significance at r(61)=.580,/?<.001. 

Next, a series of six regressions were conducted to determine if the variables of the study were significant 
predictors of writing ability. The regression results consistently indicated that when the follow-up estimate of 
writing ability was employed as the dependent variable, the initial estimate of writing ability and the group were 
significant predictors of the variance in writing ability with six of the variables having significant unique influence 
on writing ability. Due to space limitations we have only reported total variances of the variables, excluding 
descriptives and regression tables. In order to importance they were: composite (R 2 =. 116), initial estimate of writing 
ability scored by the online automated essay scorer (R 2 =. 202), initial estimate of writing ability scored by the human 
scorer and the online automated essay scorer (R 2 =. 205), composite and group (R 2 =. 395), initial estimate of writing 
ability scored by the online automated essay scorer and Group (R 2 =A04), and the initial estimate of writing ability 
scored by the online automated essay scorer, the human scorer, and Group (R 2 =A96). 

Finally, /-tests were conducted to determine any significant differences with regard to the scorers between 
the control group and the experimental group. The first /-test measured the control group by comparing the initial 
estimate of writing ability and the follow-up estimate of writing ability scored by the human scorer and the initial 
and follow-up estimate of writing ability scored by the online automated essay scorer. When scored by the human 
scorer the mean for the control actually declined from the initial estimate (Mean=3.59) to the follow-up estimate 
(Mean=3.44), but no significant difference was detected (/(31)=.926 p=. 361). When scored by the automated essay 
scorer, the mean for the control group increased from the initial estimate (Mean=2.44) to the follow-up estimate 
(Mean=2.50). Again, no significant difference was detected (/(31)= -.210,/?=.835). 



Table L Descriptive Statistics 







N 




Means 


Standard Deviations 




Control 


Experimental 


Control 


Experimental 


Control 


Experimental 


Human Initial 
Estimate 


32 


31 


3.59 


3.23 


1.043 


1.230 


Online Initial 
Estimate 


32 


31 


2.44 


3.52 


1.523 


1.338 


Human Follow 
up Estimate 


32 


31 


3.44 


3.65 


.914 


.839 


Online Follow up 
Estimate 


32 


31 


2.50 


4.03 


1.344 


.752 
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Table 2. Correlations of estimates of Writing Ability, Scorers, and Group membership (N=63) 







Initial 


Follow up 


Initial 


Follow up 


Group 






estimate 


estimate 


estimate 


estimate 








Human 


Human 


online 


Online 








scorer 


scorer 


scorer 


scorer 




Initial estimate 


Pearson correlation 


1 


.241 


.137 


.004 


-.162 


Human scorer 


Sig. (2 tailed) 




.058 


.284 


.974 


.205 


Follow up estimate 


Pearson correlation 




1 


.061 


.323* 


.119 


Human scorer 


Sig. (2 tailed) 






.633 


.010 


.351 


Initial estimate 


Pearson correlation 






1 


.449* 


.357* 


Online scorer 


Sig. (2 tailed) 








.000 


.004 


Follow up estimate 


Pearson correlation 








1 


.580* 


Online scorer 


Sig. (2 tailed) 










.000 


Group 


Pearson correlation 
Sig. (2 tailed) 










1 


* p<.01 



For the experimental group, when scored by the human scorer, the mean increased from the initial estimate 
(Mean=3.23) to the follow-up estimate (Mean=3.65) but no significant difference was detected (f(30)=- 1.850, 
p=A25). When scored by the online automated essay scorer, the mean increased from the initial estimate 
(Mean=3.52) to the follow-up estimate (Mean=4.03) which neared statistical significance (f(30)= -2.232, /?= 033). 

Discussion 

The statistical results have two key implications. First, the results of the human scorer are inconsistent and 
not significant in any test of single or combined independent variables. Where the human scorer is considered, 
neither the initial estimate of writing ability nor the group membership helped predict the final estimate of writing 
ability for either the control or the experimental group. When state sponsored writing exams are typically scored, 
more than one human scorer is employed to score each essay to ensure that scores are not inappropriately high or 
low. In contrast, classroom practice usually dictates that a single teacher scores each essay produced by a student. 
This study used a single highly trained, experienced Advanced Placement English essay scorer. The average English 
teacher does not have specialized training in scoring student essays. “Just as writers’ work is affected by their 
attitude to their subject matter, so too the meaning and impact of a text is controlled by readers’ purposes and any 
biases or feelings generated about the material they are reading” (Huot, 1997, p. 210). Even with specialized training 
and extensive experience, in this study, a single human essay scorer could not produce consistent scoring results. 

Nor did the human scorer’s results agree with the online automated essay scorer’s results. In fact, according to the 
human scorer, the average score of the control group declined from the initial estimate (Mean=3.59) to the follow up 
estimate (Mean=3.44) of writing ability. 

Secondly, where the online automated essay score was considered, the independent variables were all good 
predictors of performance on the final estimate of writing ability with initial estimates combined with group 
membership explaining approximately 40% of the variance even when the human scorer’s initial estimate is 
included with the previous two variables. According to the online automated essay scorer, both the experimental and 
the control groups show improvement in writing ability. Though the control group’s improvement is not statistically 
significant, the experimental group’s improvement is statically significant. Further, though the initial estimate of 
writing ability was an important predictor, group membership (i.e., control or experimental) was a much more 
important predictor. 

The results indicate that the experimental treatment of incorporating an automated online essay scorer did 
improve student writing ability in this study. Although the human scorer’s results were not statistically significant, 
the mean of the experimental group did increase from the initial estimate to the follow-up estimate of writing ability. 
The online scorer’s results more strongly indicated improvement between the initial estimate and the final estimate 
of writing ability. The increase of the mean did reach statistical significance thus indicating an improvement 
according both scoring methods. These results show that the human scorer was unreliable in this study while the 
online scorer was very consistent. In line with previous research (Straub, 1997) that demonstrated that students 
prefer comments on their writing that is specific, elaborate, and points out problems with suggestions for 
improvement, the feedback and practice provided by the online automated essay scorer appeared to have had a 
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significant impact on writing ability when integrated into the experimental school’s writing curriculum while the 
standard curriculum used by the control group did not. Therefore, the incorporation of online automated essay 
scorers into existing curriculums would provide an advantage to writing teachers and their students in improving 
standardized writing scores. The success of the online scorer in this study is consistent with research literature that 
emphasizes the importance of the use of consistent scales when evaluating writing (Hillocks, 1984; Langer & 
Applebee, 1986). The online essay scorer is not subject to the variability of the human scorer when rendering 
holistic evaluation of student performance. In addition, the handbook and suggestions for revision provided by the 
online scorer serve as instructional scaffolding (Langer & Applebee, 1986), which has also been put forth as an 
instructional model for composition instruction. 



Conclusions 

The results of this study support the integration of an online automated essay scorer because of (a) the 
reliability it demonstrated with regard to holistic scoring and (b) the statistically significant improvement in writing 
ability of the experimental group. However, this research was limited in scope. The actual instruction used by the 
classroom teacher in conjunction with the implementation of the automated essay scorer was not studied. Likewise, 
the instructional practices employed within the control school also went unexamined. The focus of this research was 
delimited to the examination of the scoring methods. In order for schools to make effective use of this expensive 
resource more data about the pedagogy with the experimental group would be beneficial. Further research should 
consider: (a) the number of times students actually used the automated essay scorer, (b) whether the students 
completed writing assignments without using the automated essay scorer, (c) what instruction the teacher provided 
prior to and after using the online essay scorer, and did topic selection influence student motivation and/or 
performance? How can the improvements seen in the experimental group be increased? In addition, some debate 
among researchers in writing assessment exists about the validity of holistic scoring, or “the value of the judgment 
given by a rater” (Huot, 1990, p.205), such as Criterion. Would other methods of assessment produce similarly 
statistically significant results? The use of scoring for (a) development and focus or (b) correctness and efficiency 
rather than holistic scoring has been debated as means of enhancing validity in scoring writing samples (Huot, 

1990). 
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Self-Regulation of Learners in an Asynchronous University Math 

Course 
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Virginia Tech 

Introduction 

In recent years a new model of course design has emerged which has become known as the 
“emporium model” (Twigg, 2003, p. 34). Synchronous class meetings do not exist for courses using the 
emporium model. Learners enrolled in an emporium-designed course work on most of their assignments 
asynchronously. Learners choose from a selection of online materials and one-on-one assistance in a 
computer lab. Some online activities may be accessed from any computer with access to the Internet and 
some activities are restricted to access from a central computer lab. The emporium model has been adopted 
at several institutions as the redesign option for mathematics courses (’’The Learning MarketSpace," 2005). 
Yet, investigations into the needs of the learners in these environments have, for the most part, eluded 
publication. Given that in an emporium-designed course students are expected to manage their own 
learning, academic self-regulation is an area that must be considered. Academic self-regulation refers to 
the degree to which learners are “metacognitively, motivationally, and behaviorally active participants in 
their own learning process” (Zimmerman, 1989, p. 329). Self-regulation has been identified as an 
important skill for learners in emporium designed math courses (Hodges, 2006). This study was conducted 
to address the lack of research on emporium-designed courses, and the need for self-regulation research 
called for in recent publications (e.g. Hodges, 2005; van Gog, Ericsson, Rikers, & Paas, 2005; Whipp & 
Chiarelli, 2004). The research question guiding this investigation was, “What strategies do learners use to 
regulate their learning in an emporium-designed mathematics course?” 

Participants 

Seven individuals (4 female, 3 male) participated in this study conducted during the spring 2006 
semester at a large mid- Atlantic university. All of the participants identified their ethnicity as White or 
Caucasian and their academic level as freshman. One female participant was 19 years old. Each of the 
other participants was 1 8 years old. The learners were enrolled in an emporium-designed, freshman-level, 
matrix algebra course. Academic majors of the participants included: mathematics (1), general engineering 
(4), mining engineering (1), and material science (1). The participants were solicited via email during the 
first week of the semester. An email invitation was sent to each of the 602 learners enrolled in the course. 

The participants’ experience with online courses was varied. Some participants had taken online 
courses previously and some had not. Some were taking online courses at the same time they were taking 
the course used in this study. One participant was repeating the course. 

The Research Context 

The context for this study was a 2-credit, emporium-designed, matrix algebra course. The 
majority of students enrolled in the course are engineering majors, but students majoring in mathematics, 
physics, chemistry, and other sciences must also take the course. Students attend a one-time, face-to-face 
introduction to the course during the first week of the semester. In that introduction the course facilitator 
explains that there are weekly quizzes, periodic tests, and a final exam. The quizzes and tests have set 
deadlines throughout the semester, but, working within those deadlines, students are free to schedule their 
time for the course, as they want. Various resources are available to support the students in their learning 
and the course facilitator highlights those during the course introduction. Resources available to the 
students include: web-based video lectures, practice quizzes, an online book referred to as lesson pages , 
and the human Instructional Assistants available in the computer lab. Each weekly quiz counting toward 
the students’ semester grade is a four-question quiz. The practice quizzes also consist of four questions. 
Instructional Assistants are available most days of the week from 10:30 AM to 10:30 PM and no 
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appointment is necessary. Instructional Assistants do not offer extended, one-on-one tutoring sessions, but 
they do offer just-in-time guidance on specific math questions. 



Research Design, Data Collection, and Procedures 

This research was conducted as what Merriam (1998) refers to as a “basic qualitative study” (p. 

1 1). It is a partial replication of a study by Whipp and Chiarelli (2004) involving learners in a web-based 
course. Data was collected through guided journal entries, one-on-one interviews, and artifact collection. 
Data was collected for this study throughout the spring 2006 semester and again in early 2007. 

Participants were asked to complete guided journal entries during the first, sixth, ninth, and twelfth 
weeks of the 15 -week semester. The journal entries were typed responses to specific questions supplied by 
the researcher. Journal questions were distributed in email notes and completed journals entries were 
returned to the researcher via reply email. One-to-one interviews also were conducted. All interviews 
were conducted in the computer lab used for some of the course activities. The interviews were conducted 
during the third, seventh, and thirteenth weeks of the study. An interview guide technique (Patton, 2002, p. 
343) was used to ensure a consistent line of questioning across the interviews. The audio portion of the 
interviews was recorded and transcribed for analysis. Artifacts were collected from two participants. One 
participant provided snapshots of his computer screen showing his method of scheduling his studies and 
another provided copies of a self-generated study guide. The researcher’s field notes, taken during the 
interviews, were a secondary data source. Participants also were asked to complete a follow-up guided 
journal entry in early 2007. 



Data Analysis and Findings 

The interview and journal data was analyzed using a general data reduction technique (Creswell, 
1998; Merriam, 1998). The researcher organized the data for analysis, read through all of the interview 
transcripts and journal entries making initial codes, classified all of the initial codes into categories, 
interpreted the code categories, and identified emergent themes. 

With the exception of the participant who was repeating the course, participants did not anticipate 
any difficulty with the mathematical content of this course at the beginning of the semester. Most had 
heard from peers that the course was “not bad”. At the beginning of the semester some participants were 
under the belief that they would have to teach themselves the course material. By early February the 
participants were reporting that they felt they could be successful in the course. Some participants had 
grown to appreciate the varied resources available for the course, but managing their time to complete 
required assignments in the computer lab was still a concern. 

Participants used various resources in the course, but common strategies between the learners were 
revealed. Participants worked multiple practice quizzes and used the online lesson pages to support their 
work either before or after taking practice quizzes. A general learning strategy was to begin studying no 
later than Wednesday before a 4:00 PM, Friday deadline each week. Most participants chose to study for 
this course in their dorm rooms. Assistance from other students was not typically necessary, but when help 
was needed, learners chose to ask friends rather than the staff of assistants available at the computer lab. 

The lab is located off-campus and traveling to the lab from campus was perceived as a burden. 

Individual participants took varied numbers of practice quizzes, but a trend of taking practice 
quizzes until 3 to 5 consecutive practice quizzes were completed with scores of 100% was established. For 
some, only 3 to 4 practice quizzes were needed to achieve this level of proficiency, while some reported 
taking as many as 15. This core strategy was used consistently throughout the semester. The learning 
strategy is shown in Figure 1. 
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Figure 1: Observed learning strategy used by participants 




At the end of the study the majority (5 of 7) of participants in this study felt that they were at least 
as successful in this course as they would have been in a traditional face-to-face course. One participant 
commented that the format of the course “forces you to work and schedule yourself’ and another felt “a 
sense of accomplishment” in this course since he was responsible for the learning the course. 

Participants made use of various self-regulation strategies; of particular note were the goal setting 
and self-evaluating strategies. All participants set goals related to earning a good grade in the course. A 
good grade ranged from passing the course (1) to earning no lower than a B, but most considered a good 
grade to be an A. Most students tracked their progress week-to-week using the online quiz and test system 
to monitor their progress. 

Nearly one year after the study began, five of the seven participants completed follow-up journal 
entries. These final journal entries were completed in early 2007. An analysis of the final journal entries 
confirmed the findings from the earlier data sources. The general strategy shown in Figure 1 was still 
reflected in the final journal entries. Two participants indicated in their final journal entries that their 
experience in the course changed their study habits in other courses. Those participants listed improved 
organization and time scheduling as benefits of taking the course. When asked if their experience in the 
course effected their opinion of online courses, four of the five participants completing the final journal 
entries indicated that they had better opinions of online courses after completing this course. The better 
opinions, however, were usually qualified with the fact that this was an introductory course and they felt 
that more difficult courses should not be offered in a similar format. When asked if the experience in the 
course had any impact on their confidence as a learner, participants responded in the affirmative. Two 
participants indicated that they were confident in their academic abilities before enrolling in the course and 
that the experience in the course reinforced that confidence. Three of the five participants completing the 
final journal entries commented that their confidence as learners had improved after completing the course. 
One participant commented, “I feel that I am able to learn in different settings. I’m not afraid to teach a 
class to myself anymore”. 
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Conclusions 



This research study was conducted to investigate the question, “What strategies do learners use to 
regulate their learning in an emporium-designed mathematics course?” Self-regulation strategies were 
identified that correspond to self-regulation activities previously determined. The strategies found in this 
study are summarized in table 1 . 

Table 1. Self-regulation strategies identified in an emporium-designed math course 



Learner Behavior in Emporium-designed 
Course 


Traditional Self-regulation Behavior 


Working practices quizzes until 3 to 5 perfect 
scores in a row 


Self-evaluating 


Taking notes 


Organizing & Transforming 


Setting a goal of getting a good grade 


Goal Setting & Planning 


Watching week-to-week grade progress 


Keeping Records & Monitoring 


Rewarding completed work with leisure activities 
(friends, games, reading) 


Self-Consequating 


Asking friends assistance 


Seeking Social Assistance 


Reviewing notes 


Reviewing Records 



Note. Traditional behaviors from “ A Social Cognitive View of Self-Regulated Academic 
Learning” by B. J. Zimmerman, 1989, Journal of Educational Psychology, 81(3), p. 337. 
Copyright © 1989 by the American Psychological Association 



Designers using an emporium-model for their courses should carefully consider how these 
strategies can be supported for, or implemented by, their learners. In addition to identifying the self- 
regulation strategies used by learners in the course, unintended, yet positive outcomes were observed. 
Participants felt that their confidence as learners was increased as a result of participating in the course. 
The majority of participants also held “better” opinions of online courses after completing the course used 
in this study. 

It should be noted that participants in this study were in math-intensive majors and most felt that 
the course content was not difficult. Participants cautioned that the emporium-design format may not be 
appropriate for courses with content perceived as difficult. 
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Abstract 

The National Center for Manufacturing Education (NCME) in partnership with the Quality Engineering 
Technology (QET) Department received a NSF-ATE project grant in August 2003 to develop and test a hybrid 
instructional delivery methodology. The design uses small group activity-based instructional materials developed 
under previous grants in conjunction with supportive web-based content and learning objects for the individual 
online component. This allows face-to-face interaction to occur despite the groups’ working at different locations 
and times. Web-based supplemental instructional materials and learning objects created and under test support the 
previously developed instructional modules. 

The primary outcome of the NSF-ATE grant, A Distributed Hybrid Approach to Creating a Community of 
Practice Using NSF Funded Manufacturing Engineering Technology Curriculum Modules , is to evaluate the 
effectiveness of the delivery method as a means to increase the number of students in manufacturing-related 
programs by providing institutions, companies, and students a way to work together both onsite and online in a cost- 
effective, practical way. Of secondary importance of this study is to test Clark’s “media does not influence learning” 
statement (Clark, 1994). Ultimately achieving the grant outcomes will answer one of Clark’s fundamental evaluation 
question “Did the distance education media maximize student access to new, and/ or high quality courses and 
teaching when compared with other delivery choices? Access means increasing new groups of students and 
increasing access to the teacher” (Clark, 2000). This paper presents the updated results based on a prior report 
(Houdeshell, 2006) related to meeting the project objectives, in particular, barriers to hybrid course adoption when 
taking into consideration student perceptions and academic performance versus traditional face-to-face instruction. 

Grant Progress and Challenges 



Previous Results 

Unexpected challenges were first encountered in 2004 with the challenge of no students volunteering to 
join a “hybrid” site i.e. not coming to class but meeting with an instructor/facilitator in a small group to carry out the 
activities. The author interviewed classes as to their reluctance to be excused from coming to face-to-face classes at 
the college. Students cited “I learn a lot from other student’s questions” (Houdeshell, 2005). As a result, our external 
evaluator, Social Science Research and Evaluation Corporation (SSRE), developed a questionnaire to determine 
student’s perceptions when comparing distance, hybrid (blended) and face-to-face instruction. 

Houdeshell (2005) reported the results of the questionnaire submitted to 200 level Quality Engineering 
Technology (QET) students in three different face-to-face classes and to fifty-six students in pure distance-learning 
200 level courses. The questionnaire allowed the use of a paired data t-test to determine preferences related to 
distance learning and pure face-to-face instruction. The questionnaire was administered to 200 level distance- 
learning students enrolled in the same course as one of the face-to-face course sections and to two additional 
courses. No significant differences among the responses among the three distance education courses were observed. 
No differences were apparent for these questions except for the statement; “Being required to attend class is helpful 
in motivating me to learn the material.” The distance education students scored this as less important as a motivator 
when compared to the face-to-face students. Overall the only major benefit perceived by students for distance 
learning is convenience. 

In order to meet this challenge the Principal Investigators in conjunction with the QET department split 
courses that have designated lecture-laboratory components into two separate courses, for example the 3 credit hour 
course, QET 201 Statistical Process Control (SPC) became 2 courses QET 201 SPC, 2 credit hours, a two-lecture 
hour course, and co-requisite QET 181 Laboratory for SPC, 1 credit, two-laboratory hours per week course. With 
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this course combination a variety of options are possible: Offering totally face-to-face, offering the face-to-face 
laboratory course (at the college and offsite) with a distance-learning lecture class (hybrid) or offering a pure 
distance-learning experience. Table one outlines the possible course combinations under a hybrid or blended mode. 

Table 1. 



Hybrid Course Offering Combinations 



Course 


Face-to-Face 


Web 


Day 


Eve 


Sat 


Off site 


QET 201 




X 










QET 181 


X 




X 


X 


X 


X 



Using the same methodology and questions additional data was collected over the past year from classes that have 
been offered in the new course combinations. These are reported and discussed in the next sections along with 
analysis of student performance. 

Course Description and Study Focus 

The primary courses at part of the current study include courses with significant enrollment, two 1 00 level 
courses and one 200 level course. The 100 level “Survey of Total Quality” and 200 level “Statistical Process 
Control” courses are required course in four programs with the greatest number of students, primarily male, 
enrolling from the Industrial Engineering Technology. The “Metallurgy” course is required in four programs with 
the largest number of students coming from Mechanical Engineering Technology and manufacturing related 
programs. The day students in the Industrial Engineering Technology program are typical 18-25 years old and those 
in the evening sections typical reflect “adult learners”. Approximately six years ago these courses, involved in the 
study, evolved from a more traditional course structure to greater emphasis on constructivist learning with the 
incorporation of instructional laboratory materials developed under NSF-ATE grants by the National Center for 
Manufacturing Education. The majority of the faculty who teach these courses were involved in the development 
and testing of these materials from the beginning. The current and historic course data sets used in this study assess a 
time frame after the new materials were incorporated in face-to-face classes. No major instructional design/materials 
changes occurred during the time frame reflected in the data, only delivery methods and course organization. The 
distance education (DE) delivery modes studied over this time frame include, for the QET 101 class, a videotape 
course offered through TV Sinclair and for the QET 201 class course materials and web-based activities supported 
through a WebCT course management system. Table two provides a summary of the course information used in this 
paper. The next section of this paper focuses on determining if any statistical differences exist between student 
performance based on course and delivery mode. 

Table 2. 



Quality Engineering Technology Course Information Involved in the Current Study 



Course 

Name 


Course 

Number 


Delivery 

Mode 


Number of 
Students 


Grade 

Average 


Survey of TQ (101) and 


101-01 Day 


Face-to-Face 


219 


3.08 


Laboratory (171) 


101-50 Evening 


Face-to-Face 


196 


3.42 




101-D1 


Video Tape 


16 


3.14 




101-49 


Virtual 


10 


3.67 




171-01 Day 


Face-to-Face 


10 


3.64 


Metallurgy (132) and 


132-01 Day 


Face-to Face 


109 


3.44 


Laboratory (173) 


132-50 Evening 


Face-to Face 


162 


3.45 




132-49 


Virtual 


6 


3.33 




173-01 Day/Eve 


Face-to Face 


6 


3.50 


Statistical Process Control 


201-01 Day 


Face-to Face 


107 


3.00 


(201) and Laboratory (181) 


201-50 Evening 


Face-to Face 


109 


3.25 




201-TC 


Web DE 


15 


3.42 




201-49 


Virtual 


13 


2.67 




181-01/50 Day 


Face-to Face 


13 


2.91 
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Analysis of Student Performance 

The analysis of the student performance addresses the question “Does the delivery mode affect student 
performance and retention in the three course areas?” To answer this question the author used Chi-square goodness 
of fit tests to determine if the grade and withdraw data reported from the Registrar’s office indicated any statistical 
differences. In this case the distribution of letter grades and withdraw numbers formed the basis for the distribution 
cells. This is important because just because a course has the same overall course average the mix of letter grades 
and a high drop rate in one course could reject the hypothesis that they are the same distribution. For example, the 
Chi-squared statistical testing the differences in the distributions between the day and evening sections of QET132 
indicate a probability of 0.10, that by chance alone these distributions are the same even though they had the same 
overall averages. A review of the actual grades, illustrated in Table 3, indicated that the day section had a spread of 
grades with lots of A’s and the evening only reporting A’s and B’s and a larger percentage of withdraws. 

Table 3. 



Distribution of Grades by Percent in Metallurgy (QET 132) 



Section 


A’s 


B’s 


C’s 


D’s 


F’ 


W’s 


Day - 01 


72 


15 


6 


1 


1 


5 


Eve - 50 


79 


9 


0 


0 


0 


11 



The author also observed the same pattern in the comparison of the QET 201 day sections and the QET 201 web 
based course sections where the TC sections have A’s and B’s and lots of withdraws. Table four reports the Chi- 
Squared results when comparing courses based on, day versus evening sections and delivery mode. 

Table 4. 



Grade Distribution Comparisons between Quality Engineering Technology Course Sections 



Course and Section Comparisons 


df 


c 2 


P 


Action 


101-01 vs. 101-50 


5 


16.61 


0.0053 


Reject Same Distribution 


101-01 vs. 101-D1 


5 


3.54 


0.6171 


No Significant Difference 


132-01 vs. 132-50 


5 


9.12 


0.1042 


No Significant Difference 


201-01 vs. 201-50 


5 


14.64 


0.0120 


Reject Same Distribution 


201-01 vs. 201-TC 


5 


15.78 


0.0075 


Reject Same Distribution 


201-01 vs. 201-49/181-01/50 


5 


2.50 


0.8688 


No Significant Difference 



An observation indicates that the night and distance students typically are more successful focused based 
on grade distribution pattern. It appears that if they are not receiving an A or B then tend to withdraw at a higher 
rate. A second observation is that the delivery mode does not appears to have a significant impact on student 
performance in support of Clark’s stated postulate that media does not influence learning (Clark, 1994) . 

Additional Student and Faculty Perceptions 

Over the past couple years data additional students have been surveyed in both face-to-face, and hybrid 
offerings as to their preferences for learning. These paired data t-test results displayed in Appendix A provides 
insight into the student’s perceptions as to the benefits of face-to-face instruction versus pure distance delivery and 
face-to-face instruction versus hybrid delivery. Students enrolled in face-to-face, distance education, and hybrid (pre 
and post completion) defined the major perception outcomes. First, distance education students prefer face-to-face 
except for the convenience of distance education classes. They also perceived that it would be easier to work with 
other students within a face-to-face environment for all other questions. Face-to-face students prefer face-to-face 
instruction to pure distance education and inconclusive results for students preferences concerning pure face-to-face 
over a hybrid delivery. This could be based on the students’ comfort level with the content material and preferred 
learning style (Martinez, 2002). 

The group of distance-learning students answering the same questionnaire concurred with the face-to-face 
student results with the exception of two questions. The distance-learning students perceived, at a very significance 
level, that it would be easier to get questions answered within a face-to-face environment and also concluded 
participating in distance learning classes was more convenient than face-to-face classes. The students also completed 
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questions related to their learning styles and social interaction, No differences were apparent for these questions 
except for the statement; Being required to attend class is helpful in motivating me to learn the material. The 
distance education students scored this as less important as a motivator when compared to the face-to-face students. 

Faculty perceptions follow student perceptions and for those results of the analysis for both students and 
faculty are found in Appendix A. The answers to faculty questions that did not apply to students typically reflect the 
amount of effort required in the classroom or on-line. Face-to-face classes are perceived to require the least amount 
of preparation and hybrid and distance education classes require the most. 

Summary 

The impact of the documented project adds to the scope of body of knowledge related to barriers to 
adoption. One of the most commonly cited reasons for blending is more effective pedagogical practice. “BL is the 
combination of instruction from two historically separate models of teaching and learning: traditional F2F learning 
systems and distributed learning systems” (Graham, 2004). This approach ends the isolation of pure distance 
education student by providing a combination of group face-to-face and individual asynchronous learning 
opportunities. The authors concur that well developed hybrid or blended courses based on sound instructional 
delivery are effect in student learning and retention. Since the original grant was funded in 2003 many authors have 
touted hybrid and blended instruction (Giguere & Houdeshell, 2005; Stinson, 2004; Swenson & Evans, 2003). The 
challenge is finding methods and practices that aid in overcoming the barriers to adoption based on administrative 
policy, faculty reluctance and student misconceptions. Also, additional studies that explore other barriers such as 
access to technology, course creation and/or adaptation issues, and integration and planning also need to be 
explored. 

This research is supported in part by the National Science Foundation under DUE-0302574. Any opinions, 
findings, and conclusions or recommendations expressed in this material are those of the author and do not 
necessarily reflect the views of the National Science Foundation. 
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Appendix A 

Significance comparisons based on FtF versus hybrid or web delivery. 



Faculty Perception Response 



How hard will it be to respond to questions? 



Delivery 


Average 


FtF 


4.27 


Hybrid 


4.00 


Web** 


3.32 


How hard will it be to have students work in teams? 


Delivery 


Average 


FtF 


4.41 


Hybrid** 


3.41 


Web*** 


2.36 


How hard will it be to teach the course? 


Delivery 


Average 


FtF 


4.41 


Hybrid*** 


3.45 


Web*** 


2.86 


How hard will it be for an average student 




to do well in the course? 




Delivery 


Average 


FtF 


4.05 


Hybrid*** 


3.09 


Web*** 


2.45 


How much time will students spend if 
they want to do well in the course? 


Delivery 


Average 


FtF 


3.41 


Hybrid 


3.57 



Student Response 



Receive answer to a question? 



Delivery 


Average 


FtF 


3.83 


Hybrid** 


3.22 


Web*** 


3.11 


Work with other students? 



Delivery Average 



FtF 


3.38 


Hybrid** 


2.68 


Web*** 


2.18 


Easy to learn? 



Delivery Average 



FtF 


3.23 


Hybrid** 


2.67 


Web*** 


2.43 


How convenient would it be to take 




the course in the face-to-face format? 




Distance-learning? or hybrid formats? 




Delivery Average 


FtF 


3.05 


Hybrid 


3.05 


Web FtF** 


2.18 


Web DE** 


3.80 
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Web* 


4.10 


Faculty response continued 




How much time will you spend on the course? 




Delivery 


Average 


FtF 


3.27 


Hybrid*** 


3.95 


Web*** 


4.36 


How satisfied will you be teaching the course? 


Delivery 


Average 


FtF 


4.41 


Hybrid* 


3.82 


Web*** 


2.86 



p<0.1 

p<0.01 

p<0.001 
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Cognitive presence, a sense of “being there” cognitively, has recently been considered an important factor for 
students’ engagement in e-learning. There is, however, no widely accepted quantitative measurement scale for 
cognitive presence because most studies on cognitive presence have been conducted qualitatively. Therefore, 
existing theories on cognitive sector-related research regarding cognitive presence were reviewed and a new 
measurement scale for cognitive presence was developed. This study tested reliability and validity of the 
measurement scale of cognitive presence with 305 undergraduate students. Three major factors of cognitive 
presence were perceived levels of: 1) understanding content, 2) constructing knowledge, and 3) managing 
learning resources. 



Introduction 

Cognitive presence, a sense of “being there” cognitively, has been mainly researched by constructivists who 
stated that cognitive presence reflects higher-order knowledge acquisition and application in online learning 
(Garrison, 2004). Cognitive presence is also an important factor in facilitating learners’ engagement and in 
affecting a learner’s level of achievement and satisfaction (Wang & Kang, 2005; Kang, 2005). 

Despite the importance of cognitive presence for successful online learning, there is no widely accepted 
measurement scale. Therefore, a new scale for measuring cognitive presence was developed based on the review 
of existing theories on the cognitive factors activating learners’ engagement in an online learning environment. 
Then, reliability and validity of the new scale were tested with 418 undergraduate students in an e-leaming 
environment. In this study, the developed scale was retested with 305 undergraduate students in online 
environments. 



Theoretical Background 

Studies related to online learning have begun to shift their focus to better understand the unique needs 
associated with online learners. The shift is increasing the importance of cognitive engagement in online learning 
(Richardson & Newby, 2004). One of the crucial factors for engagement in online learning is considered to be the 
level of perceived cognitive presence (Wang & Kang, 2005). In other words, engaged learners might have a 
higher level of perception of cognitive presence in an e-leaming context. 

According to cognitive engagement theory (Ryan & Patrick, 2001; Bangert-Drowns & Pyke, 2002; Wang & 
Kang, 2005), there are three characteristics that engaged learners might share: understanding, constmcting, and 
self-regulating in knowledge constmction. First, a high level of cognitive presence should be able to facilitate 
information acquisition, information transportation, and constmcting knowledge (Como & Mandinach, 1983; 
Wang & Kang, 2005). According to the cognitive engagement theory, knowledge constmction has three factors: 
information acquisition, information transformation and constmcting knowledge (Como & Mandinach, 1983; 
Wang & Kang, 2005). Based on this theory, a high level of learners’ perceived cognitive presence could facilitate 
knowledge constmction (Wang & Kang, 2005). 

Second, learners with a high level of perceived cognitive presence understand learning contents well. 

According to the cognitive engagement camp, cognitive engagement is the mobilization of cognitive strategies for 
interpretive transaction (Bangert-Drowns & Pyke, 2001). In other words, engaged learners are thought to be more 
intellectually concerned with their learning tasks intensively and extensively (Bangert-Drowns & Pyke, 2002). 



114 




Accordingly, students’ engagement entails an intrinsically motivated involvement of the integrated cognitive 
process (Kearsley & Schneiderman, 1998). 

Third, learners who perceive a high level of cognitive presence manage learning resources freely. Since 
cognitive engagement is considered a core variable in a well-developed self-regulating learning process, engaged 
learners will be able to manage well resources, environment and performance (Mckeachie et al., 1986; Pintrich & 
De Groot, 1990). Resource management involves the process of developing well-defined goals and scheduling the 
course to achieve the best results. Environment management is the development of a physical setting that is 
helpful to learners. Performance management includes self-effort, self-reinforcement and persistence (Mckeachie 
et al., 1986). 

Learners’ cognitive presence activating engagement in an online environment has been investigated from 
both constructivist and cognitive perspectives. Constructivists have defined cognitive presence as the extent to 
which learners are able to construct and confirm meaning through sustained reflection and discourse in a critical 
community of inquiry (Garrison, Anderson, & Archer, 2000). They argued that knowledge is constructed by the 
circular process of triggering events, exploring, integrating, and resolving tasks during online learning (Garrison, 
2004). Cognitive presence has also been empirically validated with respect to cognitive engagement within online 
groups as providing a theoretical background for measuring the cognitive engagement of online learning (Oriogun, 
Ravenscroft, & Cook, 2005). 

The above discussion led to the extraction of three factors of cognitive presence: understanding content, 
constructing knowledge, and learning resources management. Accordingly, the operational definition of cognitive 
presence in this study was defined as the ‘perceived level of general understanding, knowledge construction, and 
learning resources management during e-Learning.’ 



Methods 

As a result of primary factor extraction of 18 items from a survey of 418 students, three dimensions of 
cognitive presence were verified. To improve reliability, this study was retested with 305 undergraduate students. 
They enrolled in an online course titled ‘Design of College Life’ in the fall semester of 2006 at a large university 
in South Korea. The class lasted for eight weeks and the cognitive presence scale was distributed to participants as 
an online survey during the fourth week. 

The three main components of cognitive presence are the level of understanding content, constructing 
knowledge, and managing learning resources. Each component has sub-components with 9 items based on a 
theoretical framework. 

First, the level of understanding the contents has three sub-components: 1) consistency between content and 
objective, 2) organization of content, 3) articulation of content (Broadbent, 1958; Gagne, Yekovich & Yekovich, 
1993). For example, the following items were used; “I could organize what I learned with diagrams and graphs,” 
“I could explain what I learned in class,” and so on. 

Second, the level of constructing knowledge has the following sub-components: 1) information acquisition, 2) 
information transformation, and 3) constructing knowledge (Corno & Mandinach, 1983; Wang & Kang, 2005). 

For example, the following items were used; “I could collect information related to the class,” “I could make a 
connection between new information and what I already knew.” 

Third, the level of managing the learning resources has sub-components: 1) time management, 2) 
performance management, and 3) environment management (Ryan & Patrick, 2001; Zimmerman, 1990; 
McKeachie et al., 1986). For example, the following items were used; “I could do assignments by making a plan,” 
“I could avoid any distractions while studying.” 

As presented above, 27 items with a five-point Likert scale were developed via an online survey. The items 
were analyzed by an expert for content validity and modified based on recommendations. The online survey was 
conducted and 305 responses were collected. Exploratory factor analysis (EFA) was performed to verify the 
emergence of the three dimensions of cognitive presence. Principal axis factoring method was used to extract 
factors. To rotate factors, direct oblimin rotation method was used. 

Results 

The results of EFA with 27 items yielded three factors: understanding content, constructing knowledge, and 
learning resources management. The reliability of these factors with Cronbach’s coefficient alpha yielded .844, 
.809, .640, respectively. Six items showing loading lower than .40 were removed (Thurstone, 1947). As a result, 
21 remaining items are reported in Table 1. 



115 




Table 1: Emerged factors of cognitive presence 



Main 

components 


Remaining Items 


Factor 

loading 


Cronbach’s 

alpha 


Level of 
Understanding 
the Contents 


The content of class is that I usually want to leam. 


-.402 


.844 


The content of class is what I expected. 


-.404 


I think I can understand content of class good enough to 
draw diagrams and graphs. 


-.541 


I can reorganize what I learned my class. 


-.734 


I think I can outline what I learned my class. 


-.655 


I think I can explain what I learned my class. 


-.612 


I think I can discuss what I learned my class. 


-.667 


I think I can briefly summarize learning material. 


-.723 


Level of 
Constmcting 
the Knowledge 


I usually search for extra materials related to the class. 


.450 


.809 


I feel I can select materials what I need. 


.457 


I feel I can collect the information related to the class. 


.606 


I feel I can use what I learned in the class to do 
assignment. 


.645 


I feel I deeply understand what I learned. 


.507 


I feel that I’m learning in this class. 


.584 


I’m getting new perspective through this class. 


.495 


I feel I can apply what I learned in reality. 


.471 


Level of 
Managing 
the Learning 
Resources 


I feel that I can do assignments as planed. 


.468 


.640 


I know how to be helped. 


.454 


I reorganize the material for the assignment, the course 
activity, and the discussion. 


.463 


I look for the comfortable environment that I could focus 
on my study. 


.595 


I feel that I could eliminate the obstacles that disturb my 
study. 


.494 



Discussion and Implication 

We performed a second validation process on this measurement of cognitive presence based on theory. 
According to studies on the cognitive engagement camp, cognitive presence is composed of three dimensions: 
level of understanding the content, constructing knowledge, and managing learning resources. In this study, the 
cognitive presence scale was grounded in theory and validated three constructs through statistical analysis as in the 
preliminary study. 

Consequently, this study shows that cognitive presence activating learner’s engagement consists of three 
dimensions and these could be a basis for measuring the level of cognitive presence. At present, a follow-up study 
is in progress to conduct a CFA (confirmatory factor analysis). 
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Abstract 

This qualitative study investigates students’ affective domain in an online class. Garrison, Anderson and 
Archer’s Community of Inquiry Model (2000) and Rourke et al.’s Social Presence Templates (2001) were 
used to analyze transcripts, surveys, and interviews of all participants. The results of this study illustrate 
the nature of social interaction in an online environment confirming teaching, cognitive and social 
presences go hand in hand. Recommendations for instructors to create a community of learning are 
provided. 



1. Objectives 

The purpose of this study is to investigate the role of social presence in online learning environments, 
its relationship to students’ perceptions of learning and satisfaction, and to provide some recommendations 
for the instructors and/or instructional designers in their future online course development. 

2. Theoretical Framework 

The importance of cultural and social context in constructing knowledge is recognized by many 
researchers in the field. Instructional methods from this view focus on dialogue, instructor co-learning, and 
the joint construction of knowledge. (Bonk & Wisher, 2000) 

Computer Mediated Communication (CMC) supports a collaborative learning experience at a distance 
and independence of time and space. In designing online instruction, the learner’s perspective, especially 
motivational and emotional influences on learning cannot be ignored. 

Garrison and Anderson (2003) believe that the technology of online learning made it possible for both 
private reflection and public discourse within a community of learners (p. 23). They call this the 
“ Community of Inquiry Model”. In their research on the “Community of Inquiry Model”, Garrison and 
Anderson defined it as a teacher-guided, non-authoritarian community where societal knowledge is 
revealed in an equivocal, multidisciplinary manner and its goal is to structure relationships (order) to 
achieve understanding and develop ‘rationality tempered by judgment’ (Lipman, 1991, p. 8). They listed 
three elements that interconnect and influence students’ learning: cognitive presence, social presence, and 
teaching presence. 

Cognitive presence is defined as the extent to which learners construct and confirm meaning through 
sustained reflection and discourse in a critical community of inquiry (Garrison, Anderson, & Archer, 2000, 
p. 11). Social presence is viewed as socio-emotional communication in text-based communication. 

Teaching presence is defined as the design, facilitation and direction of cognitive and social processes for 
the purpose of realizing personally meaningful and educationally worthwhile learning outcomes (Anderson, 
Rourke, Garrison & Archer., 2001). They find that instructors should bring all the elements of a 
community of inquiry in a balanced and functional relationship congruent with the intended outcomes and 
the needs and capabilities of the learners (Garrison & Anderson, 2003, p. 29). Among these three elements, 
this study is concentrated on the social presence, the socio-emotional part in text-based communication. 

There are two types of social presence; innate and cultured. In earlier stages, social presence was 
considered being inherent in the medium itself, and technologies offer participants varying degrees of 
social presence (Mclsaac & Gunawardena, 1996, p. 427). Later some scholars purported that social 
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presence can be conveyed both by the medium and by the people who are involved in using the medium for 
interaction (Mclsaac & Gunawardena, 1996, p. 427). Gunawardena and Zittle (1997) claim that 
characteristics often associated with CMC — interactivity, collaboration, and reflectivity —are not inherent 
within the medium, but can result based on design, moderator’s roles, participation patterns, and 
involvement (p. 23-24). 

In this belief, Garrison, Anderson, and Archer (2000) define social presence as the ability of learners to 
project themselves socially and affectively into a community of inquiry as “real” people through the 
medium of communication being used. Because CMC would be judged initially low on social presence, the 
role of the participants in compensating for lack of cues becomes more important. 

Research on distance learning has demonstrated that social presence affects not only learning outcomes, 
but also the students’ and the instructor’s satisfaction with a course. In their study of a televised classroom, 
Hackman and Walker (1990) found that system design and teacher immediacy behavior increased 
perceived social presence, which impacted student learning and satisfaction. In their study, immediate 
behavior enhanced closeness in the teacher- student relationship. 

Garrison and Anderson (2003) state that social presence needs to be established first, in order to 
sustain cognitive presence. They argue that there must be an effective presence of the remaining elements 
of a community of inquiry — teaching and cognitive, to establish the optimal level of social presence for the 
specific educational goal (p. 54). 

Richardson and Swan’s study of online education (2003) found that students with high overall 
perceptions of social presence also scored high in terms of perceived learning and perceived satisfaction 
with the instructor (p. 73). There was a strong correlation between perceived social presence and instructor 
satisfaction. Their findings correspond to Gunawardena and Zittle’ s research (1997), and confirm that 
learning is a social activity and that individuals learn more from their interactions with others than from 
reading materials alone. 

These strongly support that social presence need to be cultivated in online classes through careful 
design and strategy, and that instructors actively teach the students social practice and use of the software 
elements that help to establish social presence (Stacey & Fountain, 2001). 

3. Methods of Inquiry and Data Sources 

This study is a qualitative case study using a combination of qualitative and quantitative data collection 
methods. This is an instrumental case study, which uses examination mainly to provide insight into an 
issue or to redraw a generalization of online courses in a degree-seeking higher educational institute (Stake, 
1994, p. 437). Salkind sees the strong point of qualitative study as a unique approach with a different set of 
underlying assumptions reflecting a different worldview of how individuals and group behavior can be best 
studied (2002, p.143). This researcher strongly believes that online learning is a collaboration among the 
course instructor and the students and it could be best described as a group behavior through qualitative 
methods. 

Participants 

This 400 level English online class offered at a university in the Midwest initially had 1 8 students 
enrolled. There were 1 5 students who attended the first face-to-face session and participated in the Pre- 
survey. Two students dropped out after the first session. The demographics of these online class students 
were gathered and analyzed based on the remaining 13 students. Among them, one student dropped out 
during the middle of the semester resulting in 12 active students with successful grades at the end of 
semester. 
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Data Sources 



At the beginning of this study, the researcher attained an approval from the publishing company of “E- 
Leaming Companion: A Student’s guide to Online Success” (Watkins & Corry, 2005) for use of their 
survey, “E-leaming Readiness Self- Assessment”, and named it a “pre-survey” in this research to examine 
students’ computer skills and their computer working patterns. 

A mid-term survey was developed to measure students’ perception of interaction and social presence in 
their online class. Survey questions were constructed based on Rourke, Anderson, and Garrison’s Social 
Presence Cues in their article, “Assessing Social Presence in Asynchronous Text-based Computer 
Conferencing” (2001), and modified after seeking feedback from the instructor of the online class, where 
the researcher did her pilot study. 

The interview was a semi- structured process through the theoretical sampling method. Each face-to- 
face interview session (two) were audio taped and transcribed for further analysis and other interviews were 
done via emails. 

To triangulate survey and interview data, a focus group was conducted. The participants of this focus 
group were junior/senior level undergraduate students who showed their interests of taking online classes in 
the near future. Through discusions with the researcher for clarification purposes, the survey and interview 
questions were modified based on their feedback by the researcher . 

A “half-way survey” was developed and administered by the course instructor to assess students’ 
progress and satisfaction levels in this class and used for data analysis upon the instructor’s approval. 

The researcher attended two face-to-face class meetings during the course and did class observations. 
All online course documents and synchronous and asynchronous chats from Blackboard Learning Systems 
were examined. Three whole class live chats and each group’s weekly chats (four group’s synchronous and 
asynchronous chats) were coded and analyzed by Rourke et al’s “Social Presence Template” and its “Social 
Presence Density” formula (2001). Social presence density (SPD) is calculated as sum of raw number of 
instances divided by the total number of words. This researcher thinks SPD is more meaningful 
comparison of transcripts than the raw numbers of social presence instances. 

In addition to using multiple sources of data and setting up case study databases, evidence and 
member checks were used to increase the validity of this study. This strategy allows the researcher to make 
sure that the study is based on participant’s voices. 

4. Results 

The participants in this study were heterogeneous groups in their age, gender, student and work status 
(part time/full time) and online experience, etc. This online course was somewhat restricted in the sense of 
class meeting time. Three- time whole class live chat sessions went against the notion of distance 
education’s “any time, any where” learning. Some students complained the designated online whole class 
chats sessions which were set up during their work hours and the inflexibility of scheduling. Most of them 
seemed to get the enough interactions through the group and class chats, but some longed for more. 

Most of the social presence coding was based on synchronous chats in the course. At the beginning of 
group chats students’ interactions were primarily at the lower levels of communication, only sharing 
information and discussion dissonance. With the progress of the class most of the groups developed a 
relationship among them as in McDonald and Gibson’s study (1998) and the length of chat sessions and 
social presence density were increased. Initially most of the students did not see the importance of social 
presence, thinking their learning in this class was an individual matter even though they negotiated 
meanings and confirmed understanding throughout group chats and whole class chats. In their Half-way 



120 




survey and other postings, they expressed the value of group discussion before the whole class discussion. 
While the numbers in affective and interactive social presence increased significantly, cohesive social 
presence increased drastically. See the table below. 



Table: Analysis of Whole Class Live Chats by Social Presence (SP) Coding* 





Affective 

SP 


Interactive 

SP 


Cohesive 

SP 


Total 

SP 


First 

session 


41 


166 


63 


270 


Second 

session 


47 


159 


84 


290 


Third 

session 


55 


230 


126 


411 



* from Rourke et al (2001)’s “Model and Template for Assessment of Social Presence” 

There was some disparity among groups and its variability was hard to determine by the demographic 
information itself. This researcher interprets it to be due to the personal characteristics of the participants. 

In general female students used higher social presence cues in their chats. Also there was a discrepancy 
between students’ social presence preference in the midterm survey and their actual use of social presence 
cues in postings. This researcher interprets it as not only their lack of knowledge in social presence but also 
learner characteristics in each session. For example, the atmosphere (social presence usage) of three people 
chats was different from two people chats even in the same group. Each participant communicated 
differently to other participants depending on whom s/he was talking to, making it very unique and 
interesting for further research. 

Many students’ concerns in these synchronous chats echo with the findings of Branon and Essex’s 
findings, such as “getting students online at the same time, difficulty in moderating larger scale 
conversations, lack of reflection time for students, and intimidation of poor typists” (2001, p. 36). The 
characteristics of a text-based medium as being reflective, explicit, and precise as described by Garrison 
and Anderson (2003, p. 50) helped some in focusing and elevating the cognitive level of the exchange, but 
frustrated some who “can’t get my thoughts down fast enough”. 

5. Discussions 

Several issues emerged during data analysis. While the length of chat sessions and social presence 
density (SPD) increased over time, some students were shown as having the highest SPD in spite of their 
low utterances and passive participation in chat sessions. Those students were interestingly all males 
whose utterances stayed low, but ended up having a high SPD. Because the formula was calculated as the 
sum of the raw number of instances divided by the total number of words, the low number of utterances 
was weighed more heavily than usual. This researcher interprets that SPD as not describing the whole 
picture of online learning, and that it is inconsistent with other findings. She believes that a rich narrative is 
more valuable in describing this online class rather than the SPD itself. 

Another issue was the relationship between cognitive presence and social presence. One student had a 
low cognitive presence but a high social presence. Although in some sessions his social presence was high 
with low cognitive presence, he encouraged his group members to express their ideas and supported them 
emotionally. His contribution was valuable in a different way and would be an interesting future research 
topic. 

In this heterogeneous group, each individual’s characteristics emerged over the chat sessions. Some 
take a role of discussion leader, time keeper, lurker, poster, etc. Their chatting styles varied considerably 
from one individual to another. Some posted long messages at one time while some preferred short 
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message postings. Some were very academically oriented, focusing only on the class discussion topic and 
rushing the discussion in 30 minutes. Another group consisting of all female students was very effective in 
sharing not only course discussion topics but also their personal stories. All of them showed high SPD and 
high utterances and stayed in the chat room for over an hour. This group seemed to enjoy the small group 
live chats very much and expressed their gratitude of having small group chat session before the whole 
class chat. This researcher believes that it helps to get to know each individual and their characteristics to 
figure out the online environment. 

In general the social presence functioned to support the cognitive and affective objectives of learning 
in this online class. The findings strongly support that social presence needs to be cultivated in online 
classes through design and strategy. Some recommended strategies for instructors are: 

have a face-to-face first session to build rapport 

establish small groups early and have a small group discussion before the whole class chat 
model effective communication by inviting individual students and using humor and emoticons, 
etc. 

guide questions to keep the focus on topics 
provide timely feedback 
have a “Q & A” online forum 

have private one-on-one chat sessions with students, if necessary. 

6. Educational Importance of the Study 

Smith and Ragan (1999) believe that the traditional separation of “cognitive, affective, and 
psychomotor domains” is very much under question. They believe that any “cognitive” or “psychomotor” 
objective has some affective component to it. Rather than viewing the domains as completely separate, 
educators should strive to integrate them when they design instruction. The social presence functioned to 
support the cognitive and affective objectives of learning in this online class. These research findings and 
recommendations will be helpful in understanding the nature of social interaction in a non-verbal 
environment and how this can be utilized to create a community of inquiry. 
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Abstract: Due to the general lack of understanding about online teaching practice in the altered learning 
environments, some faculty are ill-prepared to make the shift from traditional face- to-face classroom 
setting to the online academic forum (Care & Scanlan, 2001; Lichtenberg, 2001; Palloff & Pratt, 2001). 
Therefore, this study is aimed at exploring what online instructors perceive challenges in online teaching 
based on their own experiences. The study further explored strategies of how online instructors grapple 
with the challenges in varying cases, and examined what, in their belief, causes the students’ certain 
behaviors and classroom phenomena presented in the online classroom. Based on the archive of narrative 
of online instructors, the article sorted out the instructional challenges in online instruction according to the 
framework of Berge’s four dimension of instructors’ roles: Pedagogical, social, managerial, and technical 
roles. This article concludes that online instruction can be best approached with understanding of 
challenging aspects of online teaching. This article serves as a comprehensive map to see where the 
hindrances of proceeding online instruction are located. 

Introduction 

As Garrison (2000) has observed, the challenge facing distance education theories in the 21 st century is 
to provide an understanding of the opportunities and limitation of facilitating teaching and learning at a 
distance with a variety of methods and technologies. While institution and faculty members are feeling 
pressure to offer Web-based course to meet economic and students’ demands, we have yet not developed a 
clear directions of adaptive teaching methods which play a role to familiarize online instructors how to be 
a teacher in the altered learning environments. As traditional education integrates the use of interactive, 
multimedia technologies, to enhance individual learning, the role of teacher changes from knowledge 
source to knowledge facilitator (Gunawardena & Mclssac, 2004). However, due to the general lack of 
empirical- and theory-based understanding and guidance to be knowledge facilitator, some faculty, are ill- 
prepared to make the shift from traditional face- to-face classroom setting to the online academic forum 
(Care & Scanlan, 2001; Lichtenberg, 2001; Palloff & Pratt, 2001). 

The lack of understanding of what it entails to teach courses online calls for the exploration on what 
are the critical challenges and opportunities in the development of altered instructors’ roles in online 
education. In addition, to make the exploration more plausible, this paper follows online instructors’ own 
practice projecting four different dimension of online instructors’ empirical knowledge: pedagogical, 
managerial, social, and technological roles. 

Faculty challenges in the literature have predominantly appeared as the administrative concerns to 
deter faculty from participating in distance education. The themes of challenges of faculty in online 
distance education have been clustered around high workload and the lack of time and incentives 
(Muilenburg & Berge, 2001), the lack of the relationship to tenure (Wolcott, 1993; Markel, 1999), the lack 
of expertise in the design and delivery of course materials (O’ Quinn & Corry, 2002; Anderson, et. Al., 
2001), and the feeling of isolation (Childers & Berner, 2000). There have been limited research done on 
what challenges are facing online instructors in the Web-based classrooms. The article surveys literature 
about four instructors’ roles. The empirically-proven or evolving strategies to handle the challenging 
situation will be discussed followed by the description of the challenges in literature. 

The systematic archive of lived challenges in online instruction constructed in this study makes the fact 
of online teaching classroom visible to novice as well as experienced online instructors. Therefore, this 
study is aimed at exploring what online instructors perceive challenges in online teaching based on their 
own experiences. To open to more opportunities in distance education, the study further explored strategies 
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of how online instructors grapple with the challenges in varying cases, and examined what, in their belief, 
causes the students’ certain behaviors and classroom phenomena presented in the online classroom. 

Theoretical Frameworks 

With the archive of narrative of online instructors, the article sorted out the instructional challenges in 
online instruction according to the framework of Berge’s four dimension of instructors’ roles: Pedagogical, 
social, managerial, and technical roles. 

Challenges Pedagogical Roles in Literature 

As much as the extent of change from the delivery medium which is static, asynchronous, printed, 
hypertext materials to deliver their instruction, the substantial delay in the flow of communication between 
teachers and learner (Moore& Kearsley, 1996) and the removal of physical presence has challenged online 
instructors to alter the concept of teaching. The mutual relationship between the medium of instruction and 
the role of instructors has been reconstructed in a different format from synchronous chalk and talk after 
Internet influence whole paradigms of learning and teaching in education. The changed learning concept 
demands the shift the ways in which the instructor organizes and delivers materials (Palloff & Pratt, 2001). 
Such a shift requires the instructor to be knowledgeable about skills necessary to develop the adaptive 
pedagogical strategies to accomplish online learning objectives. 

In altered learning environment, the pedagogical roles of online instructors, therefore, revolve around 
facilitating educational processes to promote students' understanding of critical concepts, principles, and 
skills (Berge, 1995). The facilitation is best achieved when the instructor understands the characteristics of 
altered learning modes and landscape. Because of the medium facilitated, the students are not only interact 
with teachers and their peers, but also with content of instruction and interface built in the medium 
(Hillman, Willis, & Gunawardena, 1994; Moore, 1989). 

The independence of study is generally recognized as a form of learning in online learning 
environments, which seems to reduce the responsibility of accomplishing learning on the instructors’ side. 
Despite the preoccupied issues of the self-directed and independent nature of online learning (Moore, 

1989), however, Garrison (1993) asserts that teachers are responsible for establishing the balance of control 
that will ensure worthwhile outcomes and continuing efforts to learn. It is more challenging when the 
teacher has to play integral role in education as a transactional process and has his and her own legitimate 
responsibility (Garrison, 2003) to link instructional strategies to relevant creativity, critical thinking, and 
cooperative learning. Such challenging role responsibilities involve in questioning, encouraging students 
knowledge building, designing a variety of instructional activities, eliciting reflection, weaving and 
summarizing discussions, offering constructive criticism, pushing to articulate ideas and exploring 
resources, and providing explanation and elaboration where necessary (Ashton et al., 1999; Berge, 1995; 
Bonk, Kirkley, et al., 2001; Mason, 1991). 

Due to the nature of independent leaning in online environments, the parameter of online instructors’ 
roles has been widened to promote the active independent learners in online environments requiring online 
instructors to reckon another pedagogical area. The instructor’s mastery should extend to identifying self- 
motivating factors and enabling self-direction of participants (Powers & Guan, 2000). The effective 
instructor enables self-motivated students to create meaning of content rather than impart the course 
content by synthesizing points which students raised, as “weavers,” or “e-moderators”, by collecting up 
statements, and by linking them to course content(Salmon, 2000; Hiltz, 1988). Such weaving comments 
supply a unifying overview, interpreting the discussion by drawing its various strands together in a 
momentary synthesis that can serve as a starting point for the next round of debate. (Feenberg, 1989). The 
pedagogical roles of instructors in online environments have added a wide range instructional tasks that 
require the sufficient level of guidance, in-depth training and reflections. 

Challenges in Managerial Roles in Literature 

The managerial role relates to class and course management ( Coppola, Hiltz & Rotter, 2002), which 
involve “agenda setting, pacing, objective setting, rule making, and decision making. (Stake, 1994). The 
managerial role has increased the complexity of teaching in online teaching because this managerial role 
involves more the management of human resources and instructional resources than synchronous chalk and 
talk instruction. 
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Human resources, such as teaching assistants, instructional designers, and members of support service 
offices, are oftentimes given to instructors to facilitate teaching. This course manager role have been 
“unbundling” by universities which are disaggregating the multiple tasks of the instructor, traditionally 
performed by one person, by assigning them to specialized teams and professionals (Paulson, 2002). 

The role of facilitating the instructional resources includes instructional materials, use of technology, 
course design, and course procedure. Online instructors should provide the clarity in coordinating courses. 
They are required to offer explicitly written target objectives of the course, expectations, requirements, 
course activities, schedule, and rubrics of the evaluation (Mason, 1991; Bonk, at. el.). 

One of the frustrating challenges in managerial roles comes from the misunderstanding of instructors’ 
intentions. Since the delivery medium eliminates the nonverbal and verbal cues and the contexts of 
communication, the misunderstanding in communication is more likely take place in online environments. 
Despite providing strong leadership and a definite structure and agenda for the course in pursuit of the 
organized, effective course management, the instructor must be aware of and responsive to organizational 
dynamic and the unexpected reactions due to the misunderstanding of the instructor’s intentions (Mason, 
1991). Studies found that well-designed course template may mitigate the managerial issues (Conrad, 2004), 
which evolve in the earlier stages of online courses before new instructors gain extensive online experience 
(Teles, 2001). The successful managerial role will result in the optimal level of learning in the learning 
community where students learn more with self-reliance, from each other, than from the instructor while 
the instructor is omni-present (Mason, 1991). 

Challenges in the Technical Role in Literature 

The technical role is an additional requirement to the conventional role of the instructors. This role 
highly demands instructors to ensure students’ access to and familiarity with the technology so that students 
are comfortable with using technology and, in turn, could concentrate on the academic tasks ( Berge, 1995). 
Online instructors are required to successfully perform of this role to decreases the degree of technical 
difficulty as indicated one of the most significant causes of students’ dissatisfaction with web-based courses 
( Hara & Kling, 2000). Technical tasks include assisting students with technology issues, addressing 
technical concerns, diagnosing and clarifying problems encountered, and allowing students sufficient time 
to leam new programs, and techniques (Ashton, Roberts, & Teles, 1999; Berge, 1995;. Bonk, Kerkeley, 
Hara, and Dennen, 2001). 

Palloff and Pratt (2001) emphasize that the instructor’s responsibility must not end the technology 
itself, pointing to more the important matter of how the instructor use it in the design and delivery of online 
courses. To perform technological role effectively, the online instructors should have a capability to 
evaluate the effectiveness of technology. Without evaluating the pros and con’s of using the up-to-date 
technology, the adoption of streaming audio and video could create the least interesting lecture which is 
impossible to sit and listen for even seemingly brief period of time. Anderson and others (2001) also note 
that the teacher is also familiar with a wealth of the digitized resources to which they can refer students for 
further individual or group study. 

Challenges in social role in literature 

Social roles in online environments are often neglected and misunderstood. Social roles of online 
instructors requires the understanding of the nature of online learning community in developing the 
friendly, social environment which is essential for online learning (Berge, 1995). Creating social, friendly 
environment in an online environment is a challenging task because of the absence of the non-verbal 
communication that occurs in the face-to-face settings of conventional education, and the reduction in the 
amount of paralinguistic information transmitted. Since the delivery channel is lean, plain text may not be 
capable of delivering the desired level of stimulation, and one may find it more difficult to express intended 
meaning or friendly tone. However, solely through this limited text, the instructor creates trusting, familiar 
social environments, communication, and interactivity. In such online environments, the issues related to 
the communication and interactivity predominantly influence social learning environments. Such issues 
include demonstrating caring, exchanging information, conveying feeling and emotion, timely response, 
stylistic communication styles, the level of formality, initiation of conversation, greeting, praise, inviting 
tones, and so forth (Tu & Mclsaac, 2002). 
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Online instructors also should be aware of the influence of the social relationship among learners in 
online learning effectiveness. The relationship among students, as called “group cohesiveness” (Berge, 
1995; Baker & Wood, 2004), needs to be in a component of course design to develop “psychological sense 
of community (Sarason, 1974;McMillan & Chavis, 1986). Sarason (1974) sees psychological sense of 
community (PSC) as the sense that one is “part of a readily available, mutually supportive network of 
relationships upon which one could depend” (p.l). Studies have found that a weak sense of social 
cohesiveness could result in increasing the drop-out rate of online students who feel isolated and stressed 
(Eastmond, 1995).. Online instructors are demanded to establish a leadership role in nurturing community 
and modeling the social roles so that instructors lead to motivate and engage students in a community of 
inquiry (Anderson, et. al., 2001). 

The social role may be difficult to define for novice instructors when they discover predominantly the 
task-oriented online students (Bonk, 2000). Conrad's (2004) study noted that inexperienced online 
instructors often lacked essential social skills and preferred personalized communication instead of taking 
leading roles in establishing a wider community. Fulfilling the complex social responsibilities of an 
instructor necessitates sustained and authentic communication between and among teachers and students 
(Kerr, 1986). Such social role is intertwined with the pedagogical role which support student cognitive 
learning processes (Baker & Wood, 2004). 

All four instructional roles — pedagogical, social, managerial, and technical — would not necessarily be 
carried out “in their entirety” by the same person. In fact, Collins and Berge thought it would be unusual if 
they were (Collins & Berge, 1995). This multidimensional role of the instructor are interconnected and 
overlapping, with different emphases at different times and in varying degrees throughout the life of the 
course (Berge, 1995). 



Method 

Narrative inquiry, a qualitative research, was used. Online instructors’ experiences are a main resource 
of this study. To find an answer to the problem of gaps between practice and theory, this study adopted 
Schon (1987)’s suggestion for research which led this study to start not by asking how to make better use of 
research-based knowledge but by asking what we can leam from a careful examination of the competence 
by which practitioners such as online instructors actually handle their practice. Following John Dewey 
(1938), Schon (1983), and Clandinin and Connelly (2000), This study focuses on personal practical 
knowledge of online instructors reflected on their experiences and embedded in their practice, in the hope 
that it would inform practice and theory. 

The primary participants of this study included three online instructors who are teaching 
undergraduates and graduates at a large, urban four-year university in Houston, Texas. The secondary 
participants were volunteered students of the online instructor participants. The pseudonyms were used for 
the participants’ anonymity. This study used the interview method combined with conversations and 
observations. 60-90 minute-interview was conducted four times to collect data with instructor participants. 
Student participants were interviewed two times for about 60 minutes. 



Results 

Findings of the study are categorized according to the Berge’ s four instructors’ roles: Pedagogical, 
managerial, social, and technological roles. 

Challenges in Pedagogical Roles 

Pedagogical lived challenges are discussed in regard to inherent limitation in text- based 
communication asynchronous communication, vulnerable autonomous learning, and the process-oriented 
content courses. 

1. Challenge in inherent limitation in text- based communication 

The inherent limitation in text- based communication challenged online instructor participants to adjust 
their pedagogical approaches to fit into the altered teaching environments. One of instructor participants 
who has been a traditional teacher for more than 20 years, relied on reading and interpreting students’ 
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visual and verbal cues. Online, however, he could not receive any of those cues in his online classroom. 
There was absence of nodding or puzzled face expression for interpreting their progress in learning. The 
lack of physical cues caused the instructor to fail to determine to what extent students understand his 
instruction. As a result, he felt frustrated and oblivious of the state of their learning. The no provision of 
the visual cues from students led him to search the different way to understand students’ responses. One of 
his solutions was monitoring the chat session which not only allowed him to monitor the status of students’ 
learning but also allowed students congregating and communicating at the certain point of time. His intent 
of this chat session was to simulate the face-to-face classroom where students meet and discuss at the 
scheduled time in one place. His methods of understanding students have since been adjusted. Another 
instructor participant stated the similar concerns. One of the aspects that she disliked about online teaching 
was the difficulty of expressing her ideas and understanding students’ thoughts through the textual form. 
She believes that learning only through the textual form inherently held limitations in terms of 
understanding the content or students. As Short, Williams, and Christie (1976) stated that one of the most 
critical aspects of the advent of new telecommunications systems is human adjustment, Web-based 
environments require the endeavor of seeking adjusted ways of delivering their instruction. 



2. Challenge in Procrastination and Vulnerable Autonomous Learning 

The narratives of this study revealed that online instruction can not solely rely on students’ autonomy. 
Students’ procrastination followed by a variety of excuses were major challenging for the online 
instructors. They believed that the flexibility of online learning would allow students to work anytime 
anywhere, but in fact, this flexibility created a negative effect; the students delayed their work to the last 
minute. This procrastination resulted in students missing deadlines, low participation on class activities, 
missing exam times, and the like. 

One of instructor participants found a way to keep students on task in activities that students had to 
participate on a regular basis. Weekly discussion board and journaling activities were used to build online 
learning routines which were difficult tasks to build since in online environments there were no scheduled 
classes and regular physical attendance. His student participants responded positively on his well- 
structured online class which kept them on task and prevent panicking experiences at last minute. 

Proactive where communication is concerned, he also initiated contact with students were reticent 
about soliciting help. He believed that monitoring students progress regularly and supporting students were 
effective way to reduce the dropout rate. 

Since the online environment has inherent limitations, such as the absence of physical and contextual 
cues, and the limitation of the technology, the online instructors are to aware of the instructor-led forms of 
instruction in the beginning of the semester until students develop their learning pace and routines. 

3. Challenge in the process-oriented content courses 

Since the instructor participants of this study were drawn from the different subject area classes, this 
study found a myriad of evidence that the effectiveness of Web-based instruction could be determined the 
kind of the subject matter of instruction. This study showed that the certain subject areas are easier to 
transfer to online mode from face-to-face instruction that others. Instructors and students in certain subjects 
courses were more struggling in online courses than other subject areas because of the conflict between the 
process of instruction and the capacity of online delivery tools. Content areas such as English, in which 
writing and reading are major learning activities, are a perfect fit with the online delivery mode. One of 
instructor participants, an English professor, believed he got more out of students online, and students 
performed better through a variety of interaction channels. Online courses provide more opportunities to 
communicate with students than face-to-face lecture-based instruction with more than 100 students. Also, 
in his online class discussion topics were as myriad and various as he wanted. His class was benefited from 
open online discussion hours not limited by traditional hours of instruction. 

However, the instructors who taught mathematics or accounting found it difficult to reach out the best 
way to deliver the content through the means of online environments. One of participants expressed this 
difficulty in her interview. “ Accounting are process-oriented courses”, she explained. Students needed to 
see the process of solving problems. One of instructor participants, a mathematics professor, believed that 
he would not benefit from the suggestions from professors in such content areas as history and English. He 
had concerns about the special characteristics of math and the capacity of technology as a delivery tool. 
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Both subject instructors found a way to show the problem-solving process through the use of presentations 
that included a simultaneous voice lecture and handwritten note-making technology. However, the courses 
required high-tech support for both teaching and learning parties. Technology often did not work as it was 
supposed to do. Even some accounting students commented that they did not understand why the school 
offered the accounting course online. 

Challenges in Managerial Roles 

This paper located challenges in managerial roles in facilitation of teaching assistants, students’ 
dissociation from the learning environments, the lack of sense of urgency, facilitation of face-to-face 
meetings, and burdening load of work. 

1. Challenge in facilitation of teaching assistants 

The narrative of one of instructor participants brought attention to the facilitation of teaching assistant. 
Since the communication in online environments is based on one-on-one interaction, when it comes to 
answering questions, there are more question entries than face-to-face students daily. Oftentimes instructor 
participants were overwhelmed by the feedbacks and responses they had to provide to students. One of 
instructors who had 130 students in her accounting class set up a system of answering students’ questions 
through the discussion board with her TAs since she believed that immediate responses were significant in 
online learning. She believed that since students transcribe their questions in writing, there is a delay of 
response and this lessens the pleasure of learning. The process of getting an answer was usually tedious. 
Sometimes the unanswered problem prevented the course from proceeding. The answer would be given to 
students after they had lost interest in the question. So she decided to facilitate TA to respond to students 
and their needs in an immediate manner. However, the interview results with students showed that the 
negative effects of the TA facilitation. Students believed that their instructor barely answered questions on 
the discussion board — even though her two TAs covered all questions students asked. It seemed that 
students preferred to receive answers directly from the instructor. Several of students even commented 
negatively about the absence of the instructor in the class. “The teaching assistants kind of teach the class.” 
Students, it seems, who are trained in the traditional classroom setting might expect instruction to be 
delivered mainly by the instructor. They want to see the dominant instructional responses and feedbacks 
provided by the instructor, even though the online environment has a totally different mode and culture 
from the traditional one. 

2. Challenge in facilitating face-to-face meetings 

One of instructor participants, an accounting professor, who were struggling to deliver the process- 
oriented courses through online created a blended approach to online learning. She understood the 
limitation of learning accounting through online so she attempted to compensate the limitation by 
instituting review sessions, so that both learning and teaching parties could seek the merits of face-to-face 
meetings. 

She also believed that even in an online environment students should have an opportunity to see the 
instructor face-to-face. The face-to-face meeting laid a foundation; students and teachers could then form 
images of each other. She assumed that his image formation made the interaction work smoothly, and it 
initiated the relationship between both parties. 

However, the face-to-face meeting resulted in the complaints from the students who could not able to 
attend on those face-to-face review days. Her intention of bettering students’ understanding of course 
materials resulted in the complaints of the unfairness from students who were not able to attend the face-to- 
face session. They felt the deprivation of privileges. The instructor felt challenged when her good intention 
turned out to be as an ineffective instructional activity. 

3. Challenge in dissociation from the learning environments 

Instructor participants realized that his students needed a guide to set the pace of online learning. Some 
students did not log on to the class site for three weeks in the beginning of the semester. As soon as the 
class began one of participants always sent email messages to each student asking them to reply to him 
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through both WebCT mail system and students’ private mail system. “For one thing, I just want to know 
that they have logged in, and that they were paying attention. And also that is one way to get them 
immediately to ask me anything that occurs to them.” He encouraged the class to ask questions through 
emails or by telephone when the need for assistance arose. He sent emails if he noticed problems with 
students. Relying on the WebCT email system is sometimes not enough to broadcast information to every 
student because some students were irresponsible. In this case, he suggests “external email, and even [the] 
telephone” would be other solutions that would bring students to online classes after monitoring their log-in 
frequency. Although he assumed that students would have enough maturity to take care of their work, some 
students need extra reminders to keep up-to-date on class information. 

One of instructor participants noticed that nontraditional students who have not been exposed to online 
environments or have not taken traditional courses for a while lack the sense of rigor and intensity 
necessary of online students. Many of nontraditional students lost their study habits when they were out of 
school. Also, their study habits in the traditional settings did not translate well in the new environment. He 
also mentioned that to be successful in online learning or in one’s profession does not involve the same 
requirements. Students who were successful in their profession were not always successful in the class. 
Students who have lost the established habits of study need to increase their efforts in order to return to the 
practice of studying. He used the regularly-held chat session to help his students too establish the pace of 
online learning; he also monitored students to see if they are keeping up with the learning events. 

4. Challenge in the lack of sense of urgency 

One of instructor participants analyzed the causes of more procrastination in online classes. Students 
enrolled in face-to-face classes had more opportunities to feel pressured to finish their work by hearing the 
instructor repeat reminders aloud and by observing their peers discuss and prepare assignments. Talking to 
peers about the assignment and their work process also provided a sense of urgency to traditional students 
in regular classes. This was the pressure arising from their learning communities. He noticed that online 
students lacked the support from instructors’ physical messages and the pressure from learning 
communities. 

5. Challenge in burdening load of work 

One of instructor participants who had 130 students felt limited and overwhelmed to answer all 
emails in a immediate manner. The feeling of guilty of not completing responding was the constant 
nuisance during the semester. The balance between other job responsibilities and her online instruction was 
her main concern while she was constantly answering students’ questions. Her negotiation with her time 
management came to the conclusion that setting a blocked time for responding and made an 
announcement about the frequency of replying answers for students. 

Students report that they check back frequently to see if there is a response. If they are left dangling for 
days on end, they lose a feeling of connection and begin to feel lost in cyberspace. This is a negative loop 
that contributes to a lessening of participation in the discussions (Markel, 2001). For instructors who have 
more than 100 students, however, it is an impossible mission to provide feedback within a 24-hour time 
span. The guidelines should be adjusted for the situation in which the online instructors are placed. 

Challenges in Technological Roles 

The discussion of challenges in technological roles in this paper include facilitation of the adaptive 
delivery method, adoption of new technology, and low technology in high tech course design. 

1. The challenges in facilitation of the adaptive delivery method 

One of the instructor participants found the facilitation of innovative technology challenging in her 
online instruction. She initially adopted the PowerPoint presentation with her voice-over into the each 
slideshow to compensate the removal of verbal instruction in her online classroom and to increase the more 
audio- sensory dimension of her instruction which is recognized to be an effective way to promote the 
learning take place. She believed that the written mode had a limited ability to draw students’ attention to 
the instructions to follow. Therefore, she decided to facilitate audio cues to deliver her instruction by 
adopting voice-over PowerPoint presentations. Despite her time and efforts to create the voice-over 
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instruction, few students listened to her PowerPoint instruction. The task-oriented online students simply 
downloaded course materials from the computer and read them at their own convenience. When online 
students were asked why they did not take of advantage of PowerPoint instruction, they stated that 
PowerPoint instruction created the atmosphere of the face-to-face classroom where they had to sit in the 
class for a long time. They chose the online classes because the class attendance and sitting time in class 
were not their learning preference and did not fit into their life. The innovative technology facilitation as 
the supplement of face-to-face instruction is best achieved when the instructor understands the 
characteristics of altered learning modes of students, and students’ learning styles and preferences. 

2. Challenges in adoption of new technology 

One of instructor participant’s use of technology in his class is an attempts to compensate for the 
removal of face-to-face interaction. In learning technologies for his online classes, Dr. Williams relies 
mostly on his self-teaching. The conferences he attends are his main sources of ideas for what kinds of 
methods and technology are available. Once he gathers the ideas on how to enhance online classes from 
conferences, he takes action without hesitation. Any initial complications in learning the use of technology 
did not stop him from continuing to develop his course in effective ways. 

He has experienced many self-taught training for technology use experiences, for example, Camtasia 
which flash movie software it captures the movement and voice on the computer screen, and voice-over 
lecture. He facilitated the Camtasia Studio add-in for PowerPoint records. This software has allowed him to 
record all multimedia elements of live PowerPoint presentation, including animations and voice narration, 
and to publish it on a Web site. However, there was no assistance to teach him. Self-training is the only 
way to master how to adopt technology for his classes. He does not have anybody to whom he could turn 
when he needs to learn new technology. 

One of the significant findings was the emergence of a community of practice among the mathematics 
faculty members in the same department who faced the common problems. The faculty members who 
taught the mathematics courses congregated informally to share the information and technology skills to 
cope with the sustained problems. The faculty members were grouped together because they had common 
difficulties, the same content area, and similar goals; similar students; and a ready-built relationship 
because there were geographically and socially closer to communicate, to leam, and to develop ideas 
without any hesitation to ask for help. The distance education administration needs to pay special attention 
to the community of practice implication, and find a way to aid and encourage the development of these 
kinds of communities through infrastructural support. 



3. Challenge in low technology in high tech course design 

Instructor participants came to realize that the innovative, high technology could be a nuisance to 
student who used their not-up-to date technology which did not have capacity to support high technology. 
Many of student participants had experienced frustration when their computers were not comparable to use 
the online course materials developed with high technology. Even if students’ computers had capacity to 
use the online course materials, the downloading high technology materials took a great amount of time on 
the student end. Even if instructors designed a course with high-tech applications, the recipients would not 
fully benefit from the effects of the technology. It is easy to fall into the illusion that expensive technology 
could solve all pedagogical dilemmas and lead us to instructional success. However, for online students 
who are mostly task-oriented, who are not up-to-date on technology, and who lack the time to attend class 
on campus, the use of time-consuming technology is not appealing. 

Challenge in Social Roles 

In this section of the paper, challenges in social roles are discussed in relation to physical dispersion. 

1. Challenges in physical dispersion 

One of instructor participants pointed that the major challenges of building social dynamics in online 
environments were largely caused by physical dispersion. In instructor participants narratives revealed that 
since online instructors taught mainly through the written format, online instructors felt limited to express 
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their enthusiasm and caring attitude toward students because the paralanguage and non-verbal cues were 
removed in their instruction online. Another reason is, they also pointed, that since online communication 
is one-to-one based interaction, there were no communication and interaction networks which were 
simultaneously and naturally built with an entire group of students when they were in face-to-face 
instruction with the entire class. Since they had limited time for online instruction, it was difficult tasks for 
them to connect socially with all of students in class. They were only able to build rapport with the limited 
number of students who actively interacted with instructors. All of instructor participants were facilitated 
a variety of the communication channels to increase the ways to communicate students including online 
chat session at 9 O’clock PM, questions through discussion board, telephone lines, online office hours and 
the like. These kinds of class activities which encouraged communication and interaction mostly failed 
unless the activities were mandatory. Most of instructors were discouraged to continue those activities or 
some were determined to change these activities as the mandatory. 

Conclusion 

The findings of study revealed that the challenges came from four dimensions of online instructors. 

The majority of challenges are attributed to physical dispersion, text-based communication, overwhelmed 
workload, and the expanded parameter of job responsibility.. The instructors were challenged to adjust their 
pedagogical roles in the text-based communication environments. The act of adjustment are required for 
both teaching and learning parties. Particularly, challenges in pedagogical roles is closely related to the 
culture of online learning environments where the pressure from peers and the sense of urgency from the 
surrounding are removed. The instructors who taught process-oriented courses such as mathematics and 
accounting had more difficult time to find the right delivery tools since the static online feature is not right 
fit for the process-oriented instruction. Challenges in managerial roles become more complex in online 
environments since the online instructors had to deal with both students and different resources to organize 
the class. They had to manage not only students but also human and technological resources. It added more 
burdens on instructors’ workload. The challenges in managerial roles calls for the instructors’ balance of 
control in accomplishing the online learning goals. Challenges in technological roles are challenged when 
the instructors had the general lack of understanding about the students’ learning styles and preferences. 

The learning and adopting new technology required more experimental and frustrating moments in online 
teaching. The findings also suggested that the infrastructure of supports for the common interests groups 
will be needed. This article concludes that online instruction can be best approached with understanding of 
challenging aspects of online teaching. Challenges in roles have interrelated and overlapping nature. 

This archive, in this sense, provide for online instructors what are expected in teaching online, and 
allow them to have better preparation in terms of coping with the challenges in online teaching roles with 
vicarious experiences provided by participants of this study. In addition, this article serves as a 
comprehensive map to see where the hindrances of proceeding online instruction are located. 
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Purpose 

This qualitative case study analyzes classroom structures that supported student autonomy in a Montessori 
classroom. Using the Axiomatic Theories of Intentional Systems (ATIS) as a theoretical model, it also explores how 
a logico-mathematical general system model can be used to characterize the configurations underlying classroom 
interaction. 



Montessori classrooms and autonomy-support 

Deci, Vallerand, Pelletier & Ryan (1991) defined autonomy or self-determination as a state where one’s 
volition was totally internalized and unaffected by external conditions. Surveys of school-age children found that 
students perceived teachers to be highly autonomy- supportive when they gave choice, had confidence in their 
ability, respected, and empathized with them (Hardre & Reeve, 2003). Autonomy-supportive social contexts were 
also found to have positive effects on children’s interest for learning, level of conceptual learning, and willingness to 
perform uninteresting activities (Grolnick & Ryan, 1987; Joussemet, Koestner, Lekes, & Houlfort, 2004). 

Montessori schools exhibit the characteristics of autonomy-support as described by the preceding studies. 
Its founder, Dr Maria Montessori, believed that education should help each child become a disciplined individual 
who is “master of himself, and can, therefore, regulate his own conduct when it shall be necessary to follow some 
rule of life.” (Montessori, 1964, p. 86). It is also undertaken with a scientific approach where children were treated 
as natural phenomenon to be observed and understood. Teachers prepare the classroom with Works or instructional 
activities, and allow students choice to engage in what interests them (Cossentino, 2006). Teachers then make 
observations, and modify the Works to maintain students’ learning engagement, and to stimulate interest in the 
Works they do not naturally choose. Even though children were given liberty to manifest themselves naturally, 
Montessori believed that they should be disciplined when they cause disruptions to learning. 

When the social context of Montessori and traditional middle schools were compared, Rathunde and 
Csikszentmihalyi (2005) found that Montessori students reported more support from teachers; more order in the 
classroom, and a greater feeling of emotional/psychological safety. They also spent more time with academic work, 
group work, collaborative learning, and individual projects; but less time in passive listening via lecturing and note- 
taking. The authors concluded that analysis of school contexts that fostered intrinsic motivation could provide 
concrete ideas for improving student engagement in public schools. 

Logico-mathematical General System Theory models 

General systems theory (GST) postulated that systems could have common characteristics and behavior 
regardless of whether these were scientific, natural or social (von Bertalanffy, 1968). Mathematical models were 
used as an “exact language permitting rigorous deductions and confirmation (or refusal) of theory” (von Bertalanffy, 
1972, p.30). Maccia and Maccia (1966)’s SIGGS theory model was developed specifically for educational 
theorizing. Thompson (2005a; 2005b; 2005c; 2006; 2007) formulated the Axiomatic Theories of Intentional 
Systems (ATIS) by improving on the consistency of SIGGS nomenclature, and converting SIGGS hypotheses into a 
set of axioms or assumptions underlying general systems behavior. ATIS basic properties describe system 
characteristics such as the number of system components (size), and the number of affect relations or connections 
between system components (complexity). Structural properties characterize the configuration of an affect relation 
set, while dynamic properties describe event changes that occur within a system, or its transactions with the 
negasystem (environment outside system boundaries). 
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Cossentino (2005) found Montessori classrooms to be characterized by discemable rituals that teachers use. 
Using the ATIS theory model, this study attempted to describe these rituals by measuring nine general system 
properties from the interactions between teachers and students, and triangulating these with qualitative and survey 
data to determine the structural configurations that support student autonomy in a Montessori classroom. 

Method 



Subjects 



The study was conducted in an upper elementary (fourth to sixth grade) classroom of a Montessori school 
located in the state of Indiana. It consisted of 28 students, ranging from 9 to 1 1 years old, a Montessori-certified 
head teacher, and two assistant teachers. 



Classroom context 

In each 16-week session, students completed mandatory number of Works which include research projects; 
book reports; workbooks in Math and flash card drills. Teachers made a punch on individual cards that tracked their 
progress when an item of their required Works was completed. A typical day in the school was from 8:30 a.m. to 
3:30 p.m. The morning started with students working on a Head Problems worksheet that consisted of math and 
logic-related problems designed by teachers. When this was completed, students moved into the Morning Work 
Period where they were free to work individually or collaboratively on their Works. About half an hour before 
recess at noon, students re-gathered to present their answers for the Head Problems. After lunch at 1 p.m., students 
attended Spanish or History classes and would end the day with cleanup duties at about 3 p.m. 

Procedure 

This study was first approved by the School Board of the Bloomington Montessori Association, following 
which consent forms for study participation were circulated to parents; where consent was obtained for 10 students. 
Ten one-hour observations were then conducted in the classroom during April 2006 where two to three observations 
were made each week. Each observation lasted about an hour, usually between 8:30 a.m. to 9:45 a.m. This time 
segment was chosen upon consultation with teachers as they felt that the classroom activities best reflected how the 
major portion of students’ learning goals were completed in a mixed- grade Montessori classroom. 

Montessori classrooms emphasized children’s freedom to move around and work where they felt most 
comfortable. Even though the use of video equipment would have afforded greater precision for data collection, it 
was not possible to film only those subjects with parental consent without disrupting the normal operations of the 
classroom. Therefore, ethnographic field-notes were used to record the various types of interactions occurring 
between teachers and students. Cossentino (2006) employed the same data collection strategy in an effort to respect 
its classroom norms when studying work patterns in a Montessori classroom. 

To determine if the motivational style of teachers is associated with their teaching method, the Problems in 
Schools Questionnaire (SDT Website, 2006a) was administered to each teacher at the beginning of the study. The 
purpose of the questionnaire and their results were not revealed until the end of observations Teachers rated 32 
motivational strategies on a Likert scale of 1 (very inappropriate) to 7 (very appropriate). A one-hour semi- 
structured focus-group interview was also conducted with them at the end of the observations. 

Students’ level of intrinsic motivation for doing schoolwork was assessed through the Academic Self- 
Regulation Questionnaire (SDT Website, 2006b). This was administered only to the students with parental consent 
both at the beginning and end of the observations so that consistency of ratings could be verified. It consisted of 32 
questions that measured four types of motivations for doing schoolwork. Each question was scored on a Likert Scale 
of 1 (Not at all true) to 4 (Very true). 
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Data analysis 



Categorizing classroom interactions 

The constant comparative method (Creswell, 1998) was used to derive 43 common interactions that 
occurred between teachers, students, and learning resources. These were further grouped into three categories (affect 
relations): Instructional , Support, and Control. The interactions and their proposed categories were verified with the 
teachers through an interview. 

Measuring system properties 

Interactions underlying each affect relation was identified by matching of frequency and pattern 
occurrences (Yin, 2003), and then described as ordered pairs that characterize the direction of connections between 
components. It is a notational system used by the Set and Diagraph theories which underlie ATIS. The nine system 
properties for each type of affect relation (Thompson, 2006) were measured as follows: 

1 . Size: The number of components in a system. 

Size, = df components of an object-set. 

2. Complexity: The total number of connections occurring between components in a system. 

Complexity -component partition, = ±f a partition, % = ( vt§o, Tvcig.J, characterized by the 
number of affect-relations. 



/ = df %n 

3. Passive Dependence: The extent to which system components are receiving connections. It is computed by first 
tabulating the number of connections received by each component, and then taking their product. 

Passive dependent-component partition measure, = a measure of receiving 

component affect-relations. 

= ± - [(£i=i nfldjty) - logilJ^i]) - n] x 100 

4. Interdependence: The extent to which components initiate and receive connections. It is computed by first 
identifying components that both initiate and receive connections. Then, the product of total paths initiated and 
received by these components is computed. 

}f[: Interdependent system measure, y S. = ±f a measure of initiating and receiving 

associated component affect-relations. 



= df [(I:=L o[log:IId = 1|) - toftlJSLJD - u] X 100 



5. Strongness: The extent to which system components are connected. It is measured by the product of total 
connections received and initiated by each component. 

Iff. Strong-component partition measure, s &. = df a measure of the degree of connected 
components. 



••WC s S) = df [(Zi=i 41 og 2 nd(v) - logi|J^|]) -n]x 100 

6. Centrality: The extent to which primary-initiating components have indirect control. It is computed by the total 
path length of connections from primary-initiating components that have a path length greater than 1 and 
comparing it \A\. Primary-initiating components are those that only initiate and do not receive connections. 

W{z Centralized-componeut partition measure, MS* = df a measure of primary- 
initiating, non-adjacent component affect-relations. 

MS =df [&i »l\ T^C*) > 1|) - log: J\|]) - n] x 100 
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7. Complete Connectivity: The extent to which system components are able to connect to other components either 
directly or indirectly. This is computed by the sum of completely connected paths occurring in the system. 

V[: Complete- component partition measure, = df a measure of pair-wise directed 

associated component affect-relations. 

= ±f [<£:=:. -Am*) > ID - -^0) o] x IDO 

8. Hierarchical Ordemess: The extent of the occurrence of a tree. A tree is an acyclic simple-graph. Except for 
the root, every connected component is directly connected to only one other component. The root is an 
initiating component (does not receive) and is directly connected to one or more other components. 

V[: Hierarchy system measure, V[( yo &), = df a measure of a w&u. 

•Vt( UQ &) = df > 1 1 x ImaxCr^ajC*) + 1)|) - log 2 |^l]) - n] x 1 00 

9. Heterarchical Orderness: A pair of components is associated if they are either adjacent or non-adjacent and have 
a directed connection between them. The length of the path between associated components is greater than or 
equal to one. Components are heterarchy-connected if they are associated and each associated pair has a two- 
way connection, or if a component is a leaf (receiving only) and associated 

VI: Heterarchy system measure, = df a measure of initiating and receiving 

associated component affect-relations ; or receiving associated (leaf) component affect- 
relations. 



^ha £ > =df KSnujiKI W0> 1 + ID - logalJ^ll) - n] x 100 



Results 



Structural differences arising from choice 

The activity pattern of Head Problems followed by the long Morning Work Period was relatively stable 
across the ten observation days. Students had to complete the worksheets prescribed by teachers during Head 
Problems but could choose the Works they wanted to do, and whether they wanted to work on them individually or 
collaboratively during the Morning Work Period. The different structural configurations with respect to Choice 
resulted in Complexity being substantially lower during Head Problems as only one Choice affect relation was 
present (See Figure 1); being that between the teacher and the problems assigned to students. This is represented as 
an affect relation set {tl, a 100} where a 100 denotes the Head Problems for that particular day. In comparison, the 
Morning Work Period found Choice affect relations occurring between each student and the Works they selected, 
thereby resulting in a higher level of Complexity. This is modeled by the affect relation set: 

{(sl,al) (s2,a2) (s3,a3) (s4,a4) (s5,a5), (s6,a25), (s7, a26), (s8,a29), (s9,a6) (sl0,a7) (sll,a8) (sl2,a9) 
(sl3,al0) (sl4,al 1) (sl5,al2) (sl6,al3) (sl7,al4) (sl8,al5) (sl9,al6) (s20,al7) (s21,al8) (s22,al9) 
(s24,a20) (s25,a21) (s26,a22) (s27,a27) (s27,a23) (s28,a28) (s28,a23)}. 

Figure 1. Comparison of structural configurations with respect to student choice of learning activity 




Interdependence was created when each student initiated and received Choice connections by selecting 
their work partners, and negotiating how they wanted to work on the project. For example, two students (s27 and 
s28) collaborated on a paper about the Olympics (a23). This is represented by the ordered pairs (s27, a23) and (s28, 
a23). They also completed other individual Works during this time i.e. (s27, a27), (s28, a28). The Morning Work 
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Period also had more Works (al to a28) that could be chosen by students, resulting in a larger Size (number of 
components in a system). As a result, the system showed more Heterarchical Orderness as Choice relationships 
were not prescribed hierarchically from one source, i.e. the teacher. 

Structural configurations with respect to Instructional affect relations 

Instructional affect relations were created between teachers and students; and between students when they 
exchanged instructional content. Differences in Instructional affect relations were found during three time periods: 
before starting Head Problems , during Head Problems and during the Morning Work Period. 

Direct instruction was the primary mode of instruction used before teachers distributed Head Problems 
worksheets for the day. Children would be gathered at one side of the classroom where the requirements of the 
worksheet were reviewed with a short refresher of related contents. This usually lasted for about 15 minutes, and is 
represented by the affect relation set: 

{(tl,sl) (tl,s2) (tl,s3) (tl,s4) (tl,s5) (tl,s6) (tl,s7) (tl,s8) (tl,s9) (tl,slO) (tl,sll) (tl,sl2) (tl,sl3) (tl,sl4) 
(tl ,sl 5) (tl,sl6) (tl,sl7) (tl ,sl 8) (tl,sl9) (tl,s20) (tl,s21) (tl,s22) (tl,s23) (tl,s24) (tl,s25) (tl,s26) (tl,s27) 
(tl,s28) (t2,sl) (t2,s2) (t2,s3) (t2,s4) (t2,s5) (t2,s6) (t2,s7) (t2,s8) (t2,s9) (t2,sl0) (t2,sll) (t2,sl2) (t2,sl3) 
(t2,sl4) (t2,sl5) (t2,sl6) (t2,sl7) (t2,sl8) (t2,sl9) (t2,s20) (t2,s21) (t2,s22) (t2,s23) (t2,s24) (t2,s25) (t2,s26) 
(t2,s27) (t2,s28) (t3,sl) (t3,s2) (t3,s3) (t3,s4) (t3,s5) (t3,s6) (t3,s7) (t3,s8) (t3,s9) (t3,sl0) (t3,sll) (t3,sl2) 
(t3,sl3) (t3,sl4) (t3,sl5) (t3,sl6) (t3,sl7) (t3,sl8) (t3,sl9) (t3,s20) (t3,s21) (t3,s22) (t3,s23) (t3,s24) (t3,s25) 
(t3,s26) (t3,s27) (t3,s28) (sl,tl) (s2,tl) (s3,tl) (s4,tl) (s5,tl) (s6,t2) (s7,t2)} 

When direct instruction was used, Instructional affect relations were initiated from the teacher to each 
student resulting in a high level of Centrality (see Figure 2) because the teacher was primary- initiating. 

Figure 2. Comparison of structural configurations with respect to instructional relations 




Despite the high level of Centrality , there was a corresponding presence of Complete Connectivity and 
Interdependence because teachers always called upon students to share their knowledge, and used their answers to 
reinforce important concepts. Comparison of observation data during this time found that about 5 to 7 students were 
called upon during these mini-lecture sessions. This is modeled with ordered pairs (sl,tl) (s2,tl) (s3,tl) (s4,tl) 
(s5,tl) (s6,t2) (s7,t2) in the affect relation set. 

Work-time during Head Problems was structurally similar to that for the Morning Work Period except that 
students had no choice over their learning activity. This is modeled by the affect relation set: 

{(al00,sl) (al00,s2) (al00,s3) (al00,s4) (al00,s5) (al00,s6) (al00,s7) (al00,s8) (al00,s9) (al00,sl0) 
(al00,sl 1) (al00,sl2) (al00,sl3) (al00,sl4) (al00,sl5) (al00,sl6) (al00,sl7) (al00,sl8) (al00,sl9) 
(al00,s20) (al00,s21) (al00,s22) (al00,s23) (al00,s24) (al00,s25) (al00,s26) (al00,s27) (al00,s28) 
(s21,t2) (t2,s21) (s22,t2) (t2,s22) (sl,t3) (t3,sl) (s2,t3) (t3,s2) (s28,t3) (t3,s28) (s28,sl0) (sl8,sl9) (sl9,sl8) 
(s4,tl) (s4,t2) (s4,t3) (s4, computer 1) (s4,sl) (s4,s2) (s4,s3) (s4,s5) (s4,s6) (s4,s7) (s4,s8) (s4,s9) (s4,sl0) 
(s4,sl 1) (s4,sl2) (s4,sl3) (s4,sl4) (s4,sl5) (s4,sl6) (s4,sl7) (s4,sl8) (s4,sl9) (s4,s20) (s4,s21) (s4,s22) 
(s4,s23) (s4,s24) (s4,s25) (s4,s26) (s4,s27) (s4,s28)} 

Centrality was also higher as compared to the Morning Work Period (See Figure 2) since a larger number 
of Instructional affect relations originated from Head Problems assignments, as shown by ordered pairs (al00,sl), 
(al00,s2). . .(al00,s28). Complete Connectivity arose out of Instructional affect relations that were formed when 
students sought help from teachers and peers to solve Head Problems. Teachers rarely told students answers directly 
but used this time for personalized coaching, thereby creating bi-directional Instructional affect relations between 
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teachers and students, as modeled by ordered pairs such as (s21,t2), (t2,s21) and (sl,t3), (t3,sl). Complete 
Connectivity was also observed between students. For example, small groups of two or three could be gathered 
around the teachers’ tables clarifying a question about the Head Problems. This is modeled by ordered pairs (sl,t3) 
(t3,sl) (s2,t3) (t3,s2) (s28,t3) (t3,s28) in the affect relation set where sl,s2, and s28 initiated Instructional affect 
relations with t3. s28 then returned to her desks and instructed another, as modeled by (s28,sl0), where s28 forms 
Instructional affect relations with slO by explaining what she learned from discussion with the teacher. Teachers 
also created opportunities for Interdependence by encouraging students to raise questions about the assigned Head 
Problems , especially when they found mistakes. They believed that this motivated buy-in for undertaking 
challenging work and allowed students to initiate suggestions for improving the contents they were working with. 

At first sight, the Morning Work Period could seem unstructured or even chaotic as each teacher and 
student seemed to be engaged in their own agendas. Five or six students could be typing up book reports on the 
computers while two or three others are working on Math workbooks at their desks. One teacher could be searching 
for a book with two students in the library; the second engaged in an individual feedback session while the third was 
grading assignments at her desk. A significant difference between Head Problems and the Morning Work Period 
was that Instructional affect relations with students originated from the different Works they chose to be engaged in, 
thereby resulting in a much lower value for Centrality as compared to the other two time periods. This is modeled by 
(al,sl). . .(a28, s28) in the Instructional affect relation set for the Morning Work Period: 

{(al,sl) (a2,s2) (a3,s3) (a4,s4) (a5,s5) (a5,s7) (a6,s6) (a8,s8) (a9,s9) (alO,slO) (all,sll) (al2,sl2) 

(al3,sl3) (al4,sl4) (al5,sl5) (al6,sl6) (al7,sl7) (al8,sl8) (al9,sl9) (a20,s20) (a21,s21) (a22,s22) 
(a23,s23) (a24,s24) (a25,s25) (a26,s26) (a27,s27) (a28,s28) (computer 1, si 2) (s 12, computer 1) 
(computer2,sl3) (sl3,computer2) (computer3,sl4) (si 4, computed) (computer4,sl5) (sl5,computer4) 
(computer5,sl6) (sl6,computer5) (computer6,s6) (s6,computer6) (s27, computed) (computed, s27) (s28, 
computer7) (computer7, s28) (booksl,sl4) (sl4, booksl) (books2,s20) (s20,books2) (tl,sl) (sl,tl) (tl,s2) 
(s2,tl) (tl,s3) (s3,tl) (t2,s6) (s6,t2) (t2,s9) (s9,t2) (t3,sl9) (sl9,t3) (t3,s20) (s20,t3) (t3,s21) (s21,t3) (s4,s5) 
(s5,s4) (s27,s28) (s28,s27)} 

Size was also larger during the Morning Work Period (see Figure 2) because individual Works and learning 
resources provide content information to support student learning. These are modeled by ordered pairs such as 
(computerl, sl2) (sl2, computerl), and (booksl, sl4) (sl4, booksl). Bi-directional Instructional affect relations 
were created as students searched for and received content information from these resources. The Morning Work 
Period was also a time where teachers met individually with students to provide feedback on their projects. These 
interactions are illustrated by the bi-directional connections occurring between teachers and students such as (t,sl) 
(sl,tl). The feedback process was described by one teacher as a time where “most of the teaching happens.” A 
meeting could sometimes take up to 45 minutes where teachers not only clarified misconceptions but also provided 
support by recommending books and resources that help students improve their projects. This thorough feedback 
process contributed to Complete Connectivity and Interdependence between teachers and students in terms of 
Instructional affect relations. 

The flexibility for students to engage in collaborative projects also created opportunities for 
Interdependence in Instructional affect relations during this time. For example, s4 and s5 chose to help each other 
check their drafts for a Science project on the topic of Convection. They were observed to be asking each other 
questions related to their drafts, thereby forming bi-directional Instructional affect relations as shown by (s4,s5) 
(s5,s4). At a computer station, s27 and s28 were found to be looking up information on the World Wide Web for a 
writing project on the Olympics, thereby create Instructional affect relations between themselves and computer 
resources. Despite increased opportunities for collaboration, Interdependence during the Morning Work Period was 
lower than the other two time periods because it was left to the sporadic intent of students. 

Structural configurations with respect to Support affect relations 

Even though Instructional affect relations were predominant, Support affect relations were found to coexist 
with them. These involved interactions related to information that was required to support instruction but were not 
instructional in nature. During Head Problems , such interactions usually involved the clarification of work 
instructions as modeled by the following affect relation set: 

{ (s 1 ,t 1 ) (s2,tl) (s4, tl) (t 1 , s 1 ) (tl,s2) (tl,s3) (tl,s4) (tl,s5) (tl,s6) (tl,s7) (tl,s8) (tl,s9) (tl,slO) (tl,sll) 
(tl,sl2) (tl ,s 1 3) (tl ,sl4) (tl ,sl 5) (tl,sl6) (tl,sl7) (tl,sl8) (tl,sl9) (tl,s20) (tl,s21) (tl,s22) (tl,s23) (tl,s24) 
(tl,s25) (tl,s26) (tl ,s27) (tl,s28) } 
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In comparison, Support affect relations were more predominant during the Morning Work Period as 
students were not constrained to doing the same activity at the same time (See Figure 3). This provided greater 
flexibility for students to engage in Support activities with each other, such as one student teaching another how to 
use the scanner. This is modeled by the interactions between si, s2, s3 and the scanner in the Support affect relation 
set during the Morning Work Period: 

{(sl,s2) (s3,sl) (sl,s3) (s3, scanner), (scanner, si) (slO,sll) (si 1, slO) (s23, s24) (s24,s23)(tl,s26) 
(tl,books4) (s26,tl) (s26,books4) (t2,s25) (s25,t2) (s7,booksl) (s8,books2) (s 14, computer 1) 
(sl5,computer2) (sl6,computer3) (sl7,computer4) (computer 1, si 4) (computer2,sl5) (computer3,sl6) 
(computer4,sl7) (books l,s7) (books2,s8)} 

Figure 3. Comparison of structural configurations with respect to support relations 




Another type of Support affect relation that occurred naturally between students would be during the 
orientation of new students. For example, when approached by new student si 1, si 0 demonstrated how to clip a 
grading sheet to the Head Problems worksheet when handing it up. s23 served as a student mentor for s24 who was 
a visitor from the lower elementary class by showing her the format for citing sources in a research report. Teachers 
also provided support to students by helping them search for additional books and resources to support completion 
of Works as shown by interactions between tl, s26 and books4. As a result, Complete Connectivity , 
Interdependence and Strongness were higher during the Morning Work Period as there was a larger number and 
more variety in the types of Support relations that were present. 

Structural configuration with respect to Control affect relations 

Even though student autonomy was a feature of the Montessori classroom, Figure 4 showed that this did 
not preclude the need for control. 

Figure 4. Structural properties relevant to control relations 
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Heterarchical Orderness Hierarchical ondemess Interdependence Passive dependence 

System Properties 



Figure 4 shows that Control affect relations were characterized by the presence of Hierarchical Orderness 
and Passive Dependence , which described connections whereby components received, rather than initiated 
relationships. Heterarchical Orderness and Interdependence were absent as Control was imposed on students 
without any means for negotiation. This is modeled by the affect relation set: 
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{(tl ,sl) (tl,s2) (tl,s3) (tl,s4) (tl,s5) (tl,s6) (tl,s7) (tl,s8) (tl,s9) (tl,slO) (tl,sll) (tl,sl2) (tl,sl3) (tl,sl4) 
(tl,sl5) (tl,sl6) (tl,sl7) (tl,sl8) (tl,sl9) (tl,s20) (tl,s21) (tl,s22) (tl,s23) (tl,s24) (tl,s25) (tl,s26) (tl,s27) 
(tl,s28)} 

Teachers initiated Control affect relations in four situations: To focus students for instructional purposes; 
for disciplinary reasons when trying to help students stay on task during the Morning Work Period and to help 
students transition between Head Problems and the Morning Work Period. 

Impact on student motivation 

Table 1 shows the average rating of respondents by motivation category. Using the internalization process 
as outlined by Deci, Eghrari, Patrick and Leone (1994), these figures showed that respondents had a greater 
tendency to undertake learning activities because they perceived some personal value and identification with the 
learning goals (Identified Regulation) rather than because they felt compelled by external factors (External 
Regulation). 

Table 1. Students’ Motivation Level for Schoolwork 



Type of motivation 


Score (Beginning) 


Score (End) 


External Regulation 


2.47 


2.54 


Introjected Regulation 


3.04 


2.88 


Identified Regulation 


3.13 


3.12 


Intrinsic Motivation 


2.15 


2.10 



Note, n = 9 



Motivation styles of teachers 

The teachers’ score on the Problems in Schools Questionnaire by motivational style is shown in Table 2. 
Table 2. Teachers’ Motivational Style 



Motivational Style 


Average Score 


Highly autonomy supportive 


46.33 


Moderately autonomy supportive 


28.67 


Moderately controlling 


23.00 


Highly controlling 


16.00 



Note, n = 3, maximum possible score = 56 



The three teachers varied in terms of their teaching experience in Montessori schools. The Head Teacher 
had 32 years of teaching experience, while the two assistant teachers had 4 and 1 year of teaching experience 
respectively. Regardless of the number of years they taught in a Montessori school, all the three teachers had the 
highest scores for the category related to Highly Autonomy Supportive and lowest scores for the Highly Controlling 
category. 



Discussion 



Autonomy support in the Montessori classroom 

The results of this study show that the instructional methods employed by teachers were congruent with the 
basic Montessori philosophy of cultivating self-mastery. Regardless of whether students sought help for 
Instructional or Support issues, teachers consistently used these opportunities to help them arrive at their own 
answers for resolving problems, resulting in the presence of Complete Connectivity throughout Head Problems and 
the Morning Work Period in terms of Instructional and Support affect relations. This underlying philosophy for 
Montessori teaching accounts for why all the three teachers rated themselves as being Highly Autonomy Supportive. 

This study also shows that Complete Connectivity in terms of Instructional affect relations could be created 
when teachers provide opportunities for students to contribute ideas and suggestions for improving the learning 
contents they are working with. This recognizes students as legitimate partners in the learning process, and impacts 



144 




their willingness to learn challenging content. In terms of Support affect relations, a free-flowing work system such 
as that of the Morning Work Period provides scope for Complete Connectivity and Interdependence between 
students. When Support affect relations are less centralized and dependent upon the availability of teachers, it allows 
them to focus on personalized instruction. The structural organization of the Morning Work Period also increases the 
Strongness of connections between system components, providing opportunities for students to foster social 
relationships and relatedness, or secure connections with people (Stipek, 2002). 

While Assor, Kaplan, Kanat-Maymon & Roth (2005)’ s study found that controlling strategies predicted 
higher levels of anger and anxiety, this study found that being autonomy- supportive does not imply the absence of 
control. On the contrary, discipline and control are periodic and necessary activities for maintaining engagement on 
learning goals. They need not necessarily result in negative affect with students if used appropriately. 

MAPS AT: Map & Analyze Patterns & Structures Across Time 

The measures of classroom structure reported in this study are new. Originally, Frick (1990) devised 
analysis of patterns in time (APT) as a methodology for measuring system dynamics - i.e., maps of temporal 
configurations or processes. APT has since been extended by Thompson (2006) to include analysis of patterns 
in configurations (APC). Configural patterns characterize structures in education - i.e., how education is 
organized, or relations between parts and whole. Together, this approach to mapping of data and methods of 
analysis is called Map and Analyze Patterns and Structures Across Time (MAPS AT). In conventional 
measurement, variables are measured separately for each case , and then some method is used to relate the 
separate measures statistically for a set of cases (e.g., correlation, ANOVA, regression, and other linear models 
approaches). On the other hand, in MAPS AT, the researcher constructs configurations. A configuration is a 
temporal or structural map that describes a given case at a given time (the case is viewed as a single system). 

Then properties of those maps are determined from either sequences of event changes (APT) or from digraphs 
of affect relations (APC). Values of these temporal or structural properties of a case is the system measure. For 
example, in the present study we determined the value of the property, interdependence, of affect relations 
during the morning work period in the Montessori classroom we observed. This is a measure of the 
configuration of affect relations. For interdependence, it is a measure of the pattern of components that both 
initiate and receive. 

MAPSAT also includes APT. Instead of a structural map, through observation we create a temporal 
map of category changes in classifications that describe changes in system events. APT was not done in the 
current study. However, in Frick (1990) one of the temporal patterns was student engagement when there was 
interactive or direct instruction. The mean likelihood of that temporal event pattern was 0.97, which means that 
if interactive instruction was occurring in a classroom, then the probability of student engagement was very 
high within each of the 25 learning environments for elementary students observed in his study. 

Axiomatic Theories of Intentional Systems (ATIS: Thompson, 2005a; 2005b; 2005c; 2006; 2007) provides 
the theoretical framework for MAPSAT. APT was designed originally to measure system dynamics, whereas APC 
has been more recently designed to measure system structure. ATIS has the potential to help us better understand 
and predict the behavior of educational systems. More information on ATIS is available in reports by Thompson 
(2005a; 2005b; 2005c; 2006; 2007). 

Limitations and suggestions for future research 

The first limitation of this study is that only one Montessori classroom was studied. Comparison and 
contrast of structural configurations between systems was not possible, thereby limiting generalizability. The second 
limitation is that only about a third of the students could be surveyed. The motivational profiles of students may not 
have been similar for students whose parents did not allow us to collect data on them. Therefore, the student survey 
results need to be interpreted with caution. 

For future research, this study could be replicated in more Montessori and K-12 classrooms and the 
structural properties compared. The relevance of other structural systems properties such as Compactness , 
Flexibleness , Weakness and Vulnerableness towards explaining classroom configurations could also be explored to 
allow additional properties of classroom structure to be understood. 
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Abstract 

Research on technology integration indicates that longer-lasting, sustainable approaches to professional 
development are needed to integrate technology effectively into the lessons of teachers. This paper presents an initial 
evaluation of a systems-based model of technology integration that uses mentoring as the main method of 
professional development. The model was put in place at an elementary school in the Southwest with 3 1 teachers 
and approximately 800 students. Results indicate that teachers under this model feel supported when using 
technology, show a tendency to increase their use of technology in the classroom, and tend to learn new ways to 
integrate technology into the curriculum. Limitations and implications for the use of the model in K-12 settings are 
discussed and suggest that the model is robust enough to address a wide variety of unique and individual barriers to 
technology integration. 

It is not altogether surprising that the presence of computers in the classroom has not naturally led to 
increased student performance. Teachers often receive improper technology training, lack the time to learn new 
technology, lack access to technology, of have poor attitudes about using technology (Ertmer, 1999; Koehler & 
Mishra, 2005; Mumtaz, 2000; OTA, 1995). Ertmer (1999) noted that teachers learning to integrate technology often 
experience these barriers in two related but different ways: externally and internally. First-order barriers are barriers 
that deal with issues external to the teacher (access to technology, time, support, resources, and training) and second- 
order barriers are barriers that deal with issues internal to the teacher (attitudes, beliefs, practice, resistance). A 
number of researchers have also noted that a major obstacle to teachers using technology for teaching subject- 
specific content is that they lack knowledge of the pedagogy associated with teaching with technology and designing 
meaningful learning opportunities that use technology (Bauer & Kenton, 2005; Hew & Brush, 2007; Hughes, 2004; 
Koehler & Mishra, 2005; Waight & Abd-El-Khalick, 2007). 

Mentoring has been found to overcome many of the barriers to technology integration in the schools 
(Frankli, Turner, Kariuki, & Duran, 2001; Gallagher, 2000; Smith & Smith, 2004). Technology mentors have taken 
a number of forms in the past decade. They have been preservice teachers guiding inservice teachers (Franklin et al., 
2001; Smith & Smith, 2004), inservice teachers guiding preservice teachers (Bullock, 2004), colleagues or peers 
(Clark, 2004; Polselli, 2002; Glazer, Hannifin, & Song, 2005), an outside expert (Lai, Trewem, & Pratt, 2002; Swan, 
Holmes, Vargas, Jennings, Meier, & Rubenfeld, 2002), or the students of the teachers themselves (Snoeyink & 
Ertmer, 2001/2). For example, Snoeyink and Ertmer (2001-2002) found that teachers used technology in ways that 
were more constructivist in nature and student-centered when the students were allowed to act as experts at using 
technology as well as with subject-matter. Regardless of who they are, mentors provide teachers with just-in-time 
support while learning to integrate technology into lessons they are actually teaching (Bullock, 2004; Lai et ah, 

2002; Whitfield & Latimer, 2003). They present teachers with a model of the various approaches to teaching with 
technology (Ertmer, 1999; Glazer et al., 2005; Lim & Khine, 2006) and provide assistance that meets the specific 
needs of the teacher (Swan et al., 2002). 

Despite its success as an approach to technology integration, mentoring is not often used as a professional 
development strategy. Some researchers (Chuang, Thompson, & Schmidt, 2003; Gallagher, 2000) suggest that 
mentoring models place too high demand on school resources such as time, money, and teacher support. Franklin et 
al. (2001) similarly note that little research exists supporting the use of mentoring as a method of technology 
professional development in schools. It is clear that while mentoring is effective at teaching teachers to integrate 
technology, it needs to be refined to improve its popularity in schools. 

The purpose of this paper is to: 1 . Present a systems-based model of technology integration that uses 
mentoring as the main approach to the professional development of teachers and 2. Present the results of using the 
model after one year in an K-5 setting. 
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The model 

The model contained in Figure 1 contains four distinct phases of technology integration set within a larger 
recursive system. The four phases of technology are: I. Initial Setup, II. Teacher Preparation, III. Curriculum Focus, 
and IV. Self-Sustaining. The system that contains and guides the four phases of technology integration was based on 
the systematic elements found in several instructional design models (Dick, Carey, & Carey, 2004; Morrison, Ross, 
& Kemp, 2001; Reiser & Dick, 1996). For example, the system begins with an analysis of needs. It then progresses 
through the establishment of goals, the selection of strategies to meet those goals, the development of 
instruction/materials, and the continuous evaluation of the model throughout. The concentric circles of the 
technology integration phases indicate that no phase is every fully complete. Rather, the concerns of each lower 
phase continue to be a concern for each higher phase, but to a lesser and lesser degree. The model has been in place 
at a K-5 school for one year. During that time, the mentor was able to move through the model into the third phase. 
The specific tasks of the mentor are contained in Figure 2, the Phase Progression Matrix. The matrix outlines the 
major activities of the mentor during each phase of technology integration. 
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Phase One: 

Initial Setup 



Phase Two: 

Teacher Preparation 




Phase Three: 

Curriculum Focus 




Phase Four: 

Self-Sustaining 




Evaluate and Revise 



Figure 1. Systems-based mentoring model containing the four phases of technology integration. 
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Mechanics 


Systems 


Culture 


Curriculum 


Phase One 

Initial Setup 


Initial setup of 
technology and 
testing that 
technology 


Develop systems 
that make 
technology use 
easiest for teachers 
and students 


Work with teaches 
who already have a 
positive attitude about 
technology. Use them 
as a bridge between 
you and reluctant 
teachers. 


Model simple yet 
effective ways of 
teaching with 
technology. 


Main Focus of the Mentor 


Phase Two 

Teacher 

Preparation 


Time will be split 
(not always 
equally) between 
troubleshooting 
technology and 
setting up new 
technology. 


Refine the initial 
systems, create 
new systems as the 
level of technology 
use increases. 


Work with teachers 
that approach you for 
help integrating 
technology. Avoid 
‘forcing’ teachers to 
use technology. 


Begin working more 
heavily with 
teachers willing to 
creating lessons that 
use technology. 


Main Focus of the Mentor 


Phase Three 

Curriculum 

Focus 


The mechanics of 
the technology 
created in earlier 
phases deals with 
troubleshooting 
and refining rather 
than creating 


Evaluate, refine, 
and/or remove 
systems from 
earlier stages. 
Create new 
systems as 
technology 
demands increase. 


Continue cultural 
practices adopted over 
the previous phases. 
Focus on teachers who 
are still reluctant to 
adopt the integration 
of technology in their 
lessons. 


Begin to monitor 
and observe the use 
of technology. Visit 
classes using 
technology and offer 
suggestions for 
improvements 
during the actual 
classroom lesson or 
activity. 


Main Focus of the Mentor 


Phase Four 

Self Sustaining 


Continue to 
troubleshoot and 
update the 
mechanics that are 
set in place 


Look to improve 
and create new and 
more efficient 
systems 


Form teacher 
technology leaders 
who will become the 
technology mentors 
for their grade level 


Offer to model 
lessons for teachers 
in the classroom and 
continue to build up 
materials in digital 
lockers. 


Main Focus of the Mentor 



Figure 2. Phase Progression Matrix. 



Method 

Participants 

Participants were 31 teachers at an elementary school (K-5) in the southwest who were provided with a 
technology mentor. Each week teachers visited a computer lab equipped with two teacher computers and 32 
computers for the students. Also available to teachers were five mobile carts containing 15 wireless laptops with 
access to the Internet and a printer. The average class size for each teacher was 22 students. Teachers also had 
available to them five carts containing 15 wireless laptops. These could be signed out ahead of time and up to two 
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carts at a time, enough for each student in a classroom to have their own laptops computer. The computers were less 
than two years old at the time of the study. Teachers had a variety of software applications available, both online and 
on the computers, including Microsoft Office. 

The teachers in the study had access to this technology the previous year, but had no mentor to support its 
use and their experience with learning to integrate the technology into the curriculum. 

Materials 

Skills Inventory. The mentor designed a Skills Inventory for determining the teachers’ initial skills with 
technology. The Inventory was a 27-item instrument on which teachers rated themselves from one to four in a 
Likert-type fashion, where a one (1) represented knowing very little about using a piece of technology and a four (4) 
represented knowing a lot. 

Teacher Interview Protocol A six-item protocol was designed by the mentor to assess teachers’ technology 
growth during the year they received a mentor. Questions on the protocol addressed teachers’ use of computers in 
the classroom, frequency of use, obstacles encountered when integrating technology, and strategies used to 
overcome those obstacles. 

Teacher Survey. A seven-item survey was designed to assess teachers’ comfort with using technology after 
having a mentor for one year. Teachers reported on the gains in skills they experienced due to the mentor and 
changes in their comfort and proficiency with technology. The survey used a Likert-type scale of four levels - SD 
(Strongly Disagree), D (Disagree), A (Agree), and SA (Strongly Agree). 

Laptop Usage Data. The use of laptop carts was tracked From September to April according to the number 
of times teachers elected to use the laptop carts in their classroom practices. 

Process 

Throughout the school year, the mentor employed the model of technology integration devoting 
approximately three months to each phase, completing the model into the third phase. The mentor met individually 
with teachers on a weekly basis and as a grade-level team once a month to plan technology- integrated activities. The 
mentor also engaged in modeling the use of technology with each teacher 5-10 times throughout the year. Teachers 
could request just-in-time support as needed throughout each school day. 

Design 

Qualitative and quantitative data were analyzed using descriptive statistics to determine the success of the 
model. The interviews were analyzed for major themes and the frequency with which those themes occurred in 
teachers’ responses to the interview items according to the recommendations of Miles and Huberman (1984). 

Results 

The results for the Skills Inventory, Teacher Interview Protocol, Teacher Survey, and Laptop Usage Data 
are presented below. 

Skills Inventory. The results of the Inventory in Table 1 indicated that the areas of most need for teachers 
were inserting images and photos into presentations and documents, using software to create graphs and calculate 
mathematics formulas, and using the online resources available throughout the district. Teachers indicated that they 
knew the least about using Excel to create formulas, simulations, and charts and graphs. 

Teacher Interview Protocol. A total of six teachers were interviewed and the interview data was analyzed 
for major themes. All six teachers stated they were using computers more this year than the year before. Four of the 
teachers also noted that their use of technology over the past year has changed and they use it more for finding 
online resources now. When asked what the biggest obstacles to integrating technology were, the major themes were 
lack of time and lack of confidence. 

Teachers were also asked how having an on-campus mentor helped them integrate computers into their 
teaching. All stated it was helpful to have a resource that could think of ideas for integrating technology and be able 
show them how to make the idea happen in an actual classroom setting. Three said it helped to have someone who 
could answer questions and troubleshoot problems. 

Teacher Survey. Twelve teachers responded to a seven- item survey (see Table 2) regarding their comfort 
with technology. Ninety percent or more responded that they agreed to strongly agree with these four statements: “I 
can usually fix computer problems”, “I can create presentations with graphics”, and “I am comfortable importing 
digital images”. Also, all 12 teachers indicated that they agreed or strongly agreed with statements about how the 
mentor helped them leam new technology skills and improve how they teach with technology. Table 2 shows the 
responses to the survey. 

Laptop Usage Data. The use of carts containing laptop computers increased steadily throughout the school 
year, beginning with 23 uses in September (one to two carts per day) and growing to 135 uses in April (four to five 
carts per day). Figure 3 contains a graph of laptop use by month for the past school year. 
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Table 1 



Resultsfor Skills Inventory^ Survey (N = 31) 



Technology / Software 


I know 
very little 


I know a 
little 


I know more 
than a little 


I know a lot 
about this 


Total 
(out of 4) 


Accelerated Reader 


-Enrolling students 


13% 


4% 


35% 


48% 


3.17 


-Producing Reports 


0 


9 


43 


48 


3.39 


PowerPoint 


-Create a slide 


17% 


9% 


13% 


57% 


3.17 


-Change font, color, etc. 


13 


9 


13 


65 


3.30 


-Add ClipArt 


13 


13 


13 


57 


3.22 


-Add digital photos 


39 


4 


22 


35 


2.52 


-Add images from the Web 


35 


13 


13 


39 


2.56 


Word 


-Change font, color, etc. 


4% 


4% 


9% 


83% 


3.69 


-Add digital photos 


39 


17 


9 


35 


2.39 


-Add images from the Web 


35 


17 


4 


43 


2.56 


-Use Drawing tools 


9 


17 


26 


48 


3.04 


Excel 


-Write formulas 


57% 


13% 


26% 


4% 


1.78 


-Create charts and graphs 


52 


17 


26 


4 


1.83 


-Create simulations 


22 


9 


22 


4 


1.65 


Search the Internet for resources 


9% 


4% 


17% 


70% 


3.48 


Outlook (send email) 


0% 


0% 


14% 


95% 


3.88 


Galileo (online benchmark 


24% 


24% 


33% 


19% 


2.48 


assessment tool) 

Knowledge Box (online self- 
contained lessons) 


-Assign Lessons 


33% 


24% 


19% 


24% 


2.36 


-Find a lesson 


38 


19 


14 


29 


2.36 


-Modify / create a lesson 


33 


14 


29 


24 


2.45 


UnitedStreaming Online Videos 


- Find a movie 


10% 


10% 


33% 


48% 


3.19 


- Download movie 


19 


0 


38 


43 


3.05 


- Play movie full screen 


19 


5 


24 


52 


3.09 


SMARTBoard 


- Making sure it is working 


29% 


19% 


38% 


14% 


2.38 


- Troubleshooting 


43 


10 


43 


5 


2.09 


- Teaching with it 


29 


14 


52 


5 


2.33 
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Table 2 



Teacher response to survey items (N = 10) 













Strongly 




Statements 


Strongly Agree 


Agree 


Disagree 


Disagree 


1 . 


I can usually fix computer problems 


6 (60%) 


4 (40%) 






2. 


I can create presentations with 
graphics 


3 (30%) 


6 (60%) 


1 (10%) 




3. 


I am proficient with formatting web 
pages 


2 (20%) 


5 (50%) 


3 (30%) 




4. 


I am comfortable importing digital 
images 


3 (30%) 


6 (60%) 


1 (10%) 




5. 


Because of the mentor I have learned 
new ways to teach with technology. 


5 (50%) 


5 (50%) 






6. 


The mentor has helped me improve 
how I teach with technology. 


3 (30%) 


7 (70%) 






7. 


I have learned new technology skills 
by working with the mentor. 


6 (60%) 


4 (40%) 








Figure 3. Laptop cart use by month. 



Discussion 

This paper presents the results of an initial evaluation of a self-sustaining systems-based mentoring model 
through the third phase of technology integration. While the data collected from teachers was limited, the results of 
the surveys, documentation of laptop usage, and interviews with teachers indicate that this model of technology 
integration resulted in an increase in teachers’ use of technology in the classroom and has improved how they use 
technology for teaching. The mentoring component of the model is most likely the reason for these successes. 
Mentoring has been found to overcome many first- and second-order barriers to technology integration (Bullock, 
2004; Franklin et al., 2001; Gallagher, 2000; Polselli, 2002; Ward, West, & Isaak, 2002). For example, Bullock 
(2004) found that teachers who received a technology mentor for 12 weeks felt less frustrated with technology and 
used technology more frequently over time. Franklin et al. (2001) found that K-6 teachers who learned to integrate 
technology with a mentor more easily overcame barriers such as finding time to integrate technology, learning to 
troubleshoot problems with technology, and learning to integrate technology into an actual classroom setting. 
Gallagher (2000) found that elementary school teachers who received classroom-based training with technology felt 
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more confident and integrated technology into their teaching strategies. The teachers in this study received a one-on- 
one mentor for the course of a year and indicated that their success using that technology in their teaching was due, 
in part, to the mentor. 

The model appears to successfully address specific first- and second-order barriers to technology 
integration that prevent teachers from using technology in the classroom - particularly issues of access, support, 
training and knowledge. Teachers reported that the mentor helped them learn to use new technology and improve 
how they teach with that technology. The mentor was also noted as helping them troubleshoot technical problems 
and to help teachers design effective lessons that incorporate technology. This finding supports researchers who 
suggested that mentoring can effectively address teachers’ need for both basic skills with technology as well as the 
pedagogy associated with teaching with technology (Ertmer, 1999, 2005; Zhao & Frank, 2003). 

A surprising finding was the dramatic increase in the use of laptops throughout the course of the year. The 
increase coincides with the end of the phase dealing with basic skills and the beginning of the phase dealing with 
integrating technology into the curriculum. This supports Hew & Brush (2007), who suggested that teachers need to 
obtain basic skills with technology before they can begin integrating it in their curriculum. It is unclear if this 
increase would have occurred naturally, meaning without the use of this model. However, teachers noted that they 
used technology more when the mentor was present than in the previous year when the mentor was not present. It is 
possible that teachers’ use of this technology would have improved without the mentor, but their experiences with 
the mentor may account for the extreme rise in technology use. 

This study is limited in several ways. The study is a snapshot of teachers’ experiences using technology 
under this model rather a before-and-after comparison. While the latter would have provided more evidence of the 
specific nature of the impact of the model, the former suggests that the model is effective at managing many 
common barriers to technology integration. Data was not collected on the barriers of time, teacher beliefs, and 
attitudes. It is not clear from the results whether these barriers were managed or not - more research on the model is 
needed to determine if the model addresses those barriers effectively. Also, the model was in the middle of the third 
phase at the time of this study. More time is needed to determine how teachers ultimately integrated technology into 
the curriculum and if they successfully developed a self-sustaining system of technology integration. 

The results also do not indicate if the quality of teacher instruction increased as a result of the model - only 
the quantity. Teachers reported that they learned new skills and pedagogy due to the mentor, but in relation to 
themselves and their own growth rather than an external focus. Future research on the model should measure the 
quality as well as the quantity of technology use among teachers as compared to other external visions of how to 
integrate technology successfully. For example, teachers’ actual use of technology could be compared to the vision 
of technology integration set by Jonassen, Howland, Marra, and Crismond (2008), who suggested that meaningful 
learning with technology occurs when students actively seek and organize information, construct new 
representations of that knowledge, and work cooperatively with peers to solve problems grounded in the context of 
the subject they are studying. 

Conclusion 

Despite the limitations of the study, the use of the model appears to have a positive impact on the use of 
technology among teachers - both in their technology skills and their teaching with technology. It is likely that this 
model would meet with success in other K-12 settings. The model employs a system, which is robust to the 
changing needs of teachers as they learn to integrate technology. This model systematically addresses the barriers 
that are unique to each given technology integration effort and designs teacher-oriented strategies for managing and 
overcoming the negative impact of those barriers. Such a model is a valuable tool for researchers, administrators, 
and teachers interested in using technology to improve the learning of students. 
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The role of perceived organizational goal structure and leadership style as 
determinants of employees’ work motivation, and intentions of innovative 

behaviors in the workplace 

Hwasook Lee 
Ewha Womans University 

Abstract 

This article examined the mediating role of employees’ work motivation between perceived organizational goal 
structure and leadership styles, and intentions of innovative behavior (N= 241). Results indicated that learning 
goal structure and transformational leadership shared a latent factor. Learning goal structure and 
transformational leadership was related to worker’s learning goal orientation, self and collective efficacy, and 
intentions of innovative behavior, whereas performance goal structure was only related to worker’s performance 
goal orientation. And these work motivation partially mediated the relationships between learning goal structure 
and transformational leadership, and intentions of innovative behavior. The implications of these findings are 
discussed. 

Keywords: goal structure, leadership style, goal orientations, self-efficacy, collective efficacy 

Introduction 

Nowadays, coping with change is essential to survive in dynamic business environment. And innovation is 
continuously required for the future of corporation. In this sense, it is necessary to consider human as core 
subjective for the sustainable growth. Therefore, it is critical to identify what motivates innovative behavior in 
the corporation. 

Since innovation is easy to fail, however, organizational culture that positively evaluates new idea is important 
employees’ innovative behavior (Jung, Chow, & Wu, 2003). When organization is not likely to accept 
innovative idea, employees hardly ever produce and implement it. For encouraging innovation, organization 
should emphasize continuously learning new things and challenging the status quo. 

However, many corporations tend to emphasize performance rather than learning and innovation which rarely 
produce outcome immediately. On the contrary, the researches of innovation have usually considered only 
innovative organizational culture (Scott & Bruce, 1994; Jung, Chow, & Wu, 2003). Therefore it is necessary to 
compare with performance-oriented organization and learning-oriented organization together. For this purpose, 
this study explored two contradicting type of organizational goal structure and leadership style, that is learning 
and performance goal structure, and transactional and transformational leadership. 

Furthermore, work motivation of employees is critical for innovation, as well as innovative organizational 
culture and leadership. Workers who has strong work motivation tend to put an effort to enhance the quality of 
product, choose more challengeable tasks, and persist longer when they face with obstacles (Schermerhom, 
Hunt, & Osborn, 2003). In the present study, individual goal orientation and efficacy was explored as work 
motivation. 

The present study proposed two primary objectives. First, this study examined the relationships between 
organizational goal structure and leadership styles, and employees’ intention of innovative behavior. Second, it 
also explored the mediating role of work motivation between these relationships mentioned above. 

Organizational Goal Structure and Individual Goal Orientation 

Individual achievement goal orientation is defined the reason why people approach, engage in and respond to 
achievement activities (Dweck & Leggett, 1988). Two contrasting goal orientations are salient in achievement 
goal theory (Dweck, 1986). First of all, learning goal oriented person is willing to learn new skill or knowledge 
and put an a lot of effort, choose challenging tasks, persist with obstacles (Ames & Archer, 1988; Ames, 1992; 
Nicholls, 1984). On the other hand, performance goal oriented person tends to focus on outperforming 
performance to others, regard an effort as lack of ability. 

This goal orientation could be influenced by situation (Ames, 1992; Ames & Archer, 1988; Button, Mathieu, 
and Zajac, 1996). For instance, if school or classroom emphasizes mastery of tasks and effort, students tend to 
pursue learning goal orientation. On the other hand, if it emphasizes competition and demonstration of ability, 
students tend to endorse performance goal orientation (Gutman, 2006). These environment are called goal 
structure by many researchers (Anderman & Midgley, 1997; Anderman & Anderman, 1998; Ames, 1992; Ames 
& Archer, 1988; Roeser, Midgley, & Urdan, 1996; Elliot & Dweck, 1988). 
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Goal structure is defined as the type of achievement goal emphasized by the prevailing practices and policies 
within a classroom, school, or other learning environments (Wo Iters, 2004). It is consisted of the task structures, 
the degree of authority, and evaluation/recognition (Ames, 1992). Researchers have discussed about two types 
of goal structure - learning goal structure and performance goal structure. Learning goal structure emphasizes 
improvement of ability and putting an effort to the challenging tasks. On the contrary, performance goal 
structure focuses on outperforming others and regards failure as the lack of ability. 

Although there are a lot of researches regarding goal structure in school environment, there is hardly ever seen 
in corporations. In this situation, there is remarkable importance to discuss about Dragoni(2005)’s research. She 
proposed that leader’s achievement goal orientation might affect work group climate (she divided it into three 
aspects - learning, performance-approach, and performance avoidance oriented culture). She also insisted that 
these work group climate could influence worker’s goal orientations. In this sense, as teachers in schools did a 
key role to give a message to students whether focusing on mastery of tasks or outperforming others, leaders in 
companies expected to perform an critical role for organizational goal structure. 

Leadership Style 

There are a lot of researches about two contrasting leadership style - transactional leadership and 
transformational leadership(Howell & Avolio, 1993). Transactional leaders expect desirable performance to 
group members. If group members meet the leader’s requirement, they can get rewards what they want. Further, 
transactional leaders tend to comment followers’ work when they make a mistake. In this sense, transactional 
leadership focused on maintaining performance, would not encourage any innovative idea. On the contrary, 
transformational leaders encourage group members with enthusiasm and vision. Bass and Avolio (1994) 
identified four components of transformational leadership: 1) charismatic role modeling, 2) inspirational 
motivation, 3) intellectual stimulation, and 4) individualized consideration. Transformational leaders also 
emphasize autonomy in follower’s work and encourage challenging the status quo (Jung & Sosik, 2002; Lim & 
Yoon, 1999). Consequently, they stimulate employees’ innovative behavior and make them achieve higher 
performance than their ability. Although these leadership styles have definitely different composition, 
transactional leadership and transformational leadership have often coexisted in corporations. According to 
Howell & Avolio (1993), many transformational leaders have also used transactional leadership in real 
companies. 

Self-Efficacy 

Self-efficacy is “a belief in one’s capabilities to organize and execute the courses of action required to produce 
given attainments” (Bandura, 1997, p. 46). In addition, individuals who have strong self-efficacy tend to put 
greater effort, persist longer when faced with obstacles (Pajares & Miller, 1994), choose more challenging tasks 
(Bandura & Schunk, 1981), adapt new technology (Hill, Smith, & Mann, 1987), and self-regulate better than 
others (Zimmerman, Bandura, & Martinez-Pons, 1992). Furthermore, self-efficacy predicts performance better 
than any other motivational construct (Aheame, Mathieu, & Rapp, 2005; Locke, Frederick, Lee, & Bobko, 

1984). 

Based on social cognitive theory, self-efficacy is formed by interaction between socio-cultural context and 
individual. In this sense, organizational goal structure positively have an influence for individual self-efficacy. 
Wood & Bandura(1989) examined goal structure and self-efficacy. They set two groups, one group got a 
message that ability can be incremental by effort, the other group got a message that ability is fixed, not changed 
by effort. As a result, the former group showed stronger self-efficacy, chose more challenging goal, and used 
more effective strategy than the latter group. Therefore, the more organization emphasizes learning and 
positively recognizes failure, the more employees have strong self-efficacy. 

Similarly, leadership style could have an implication for self-efficacy. Transformational leaders usually provide 
a model as experts, vision, intellectual stimulation, and individualized consideration with their followers. In 
other words, workers are vicariously experienced what they are going to do, verbally persuaded what they 
should do, and given positive affection for their mission from their transformational leaders. Through these 
processes transformational leaders encourage workers to have strong self-efficacy for their work. 

In addition, individual goal orientation is also closely related to self-efficacy. VandeWalle and his colleagues 
(2001) found that learning goal orientation was usually associated with stronger self-efficacy, whereas 
performance goal orientation did not have any relationship with self-efficacy. Even when learning goal oriented 
person failed in his/her job, he/she would not be frustrated and even put more efforts to overcome the fault (Bell 
& Kozlowski, 2002). 

Collective Efficacy 

Whereas self-efficacy is defined as a belief of his/her own capability, collective efficacy can be defined as a 
belief of group’s capability (Bandura, 1997). Collective efficacy has been researched in recent years because it 
has demonstrated a power to predict group’s performance effectively in various organizations such as school, 
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public organization, corporation, and even online learning environment (Aheame, Mathieu, & Rapp, 2005; 
Goddard, 2001; Goddard, Hoy, & Woolfolk-Hoy, 2000; Lee, Park, & Adler, 2005; Wang & Lin, 2006). 

This study suggested that organizational goal structure and leadership style might affect collective efficacy. 
However, evidence regarding the association between goal structure and collective efficacy is rare. In this 
situation, Porter(2005)’s approach to team goal orientation is worth to discuss. He investigated how team goal 
orientation influenced backing up behavior, performance, collective efficacy, and commitment. If team members 
have strong learning goal orientation, they are willing to help each other, show a commitment and have a strong 
collective efficacy. On the contrary, when team members have strong performance goal orientation, the results 
were confused by their performance. Although this research has an implication for goal structure, it has a 
limitation not to consider leader’s role in a group. According to many researchers of school or classroom goal 
structure, teachers play an important role to influence students’ goal orientation (Ames, 1992; Ames & Archer, 
1988; Gutman, 2006; Roeser et al., 1996). This process would be same in corporations. Dragoni(2005) also 
argued that leaders play an key role to make the group’s climate and workers usually tend to meet the group’s 
expectation. Therefore, leader’s role should be considered in organizational goal structure. That’s why this study 
transferred the classroom goal structure focusing teacher’s role into corporations. 

As well as goal structure, leadership style has an implication for collective efficacy. Transactional leadership 
emphasizes to accomplish the negotiated performance between leader and follower, whereas, transformational 
leadership emphasizes to pursue organizational value and vision. It motivates employees to interact with their 
colleagues beyond individual interest, and gives an collective identity to them (Howell & Avolio, 1993; Jung, 
Chow, & Wu, 2003). That is the critical difference between transactional leadership and transformational 
leadership (Jung & Sosik, 2002). 

Innovative Behavior 

Innovative behavior is defined that “taking initiative in improving current circumstances; it involves 
challenging the status quo rather than passively adapting present conditions” (Crant, 2000, p.436). 
Organizational culture that emphasizes learning positively affects organizational innovation (Bates, 2004). One 
of hypotheses in this study is that organizational learning goal structure might have an implication for 
employee’s innovative behavior. Also, leadership style could have an association with innovative behavior. For 
instance, Jung and his colleagues (2003) found that transformational leaders motivated followers to question 
about assumption, and access the problem with new perspective. That is, transformational leadership positively 
affects worker’s innovative behavior. 

Besides, work motivation might be an important role to provoke innovative behavior. Learning goal oriented 
people are willing to learn and choose difficult tasks that make them develop their ability. They don’t care about 
the failure. Whereas performance goal oriented people are always interested in demonstrating their ability to 
others (Ames & Archer, 1988; Ames, 1992; Nicholls, 1984). Therefore, employees’ goal orientations affect their 
innovative behavior. 

Second, workers’ efficacy could perform a critical role for their innovative behavior. Most of the time, 
individual usually considers the probability of success when they think about innovative behavior (Morrison & 
Phelps, 1999). Efficacy is a belief for future behavior and a person who has strong self-efficacy tends to take 
challenging tasks. Parker, Williams, & Turner (2006) studied that people who have strong self-efficacy tend to 
convince themselves that they have the ability of innovation, and finally challenge the status quo. 

Although individual initiate innovative behavior, he/she could not complete their innovation without 
colleague’s help. On the other hand, employees have a conservative view and do not like change though 
organization drives to innovation. In this sense, it is required to consider individual innovative behavior and 
group innovative behavior separately. 



Method 



Sample 

The sample consisted of 241 employees from 17 companies, Korea. Although 278 employees participated, 37 
employees were excluded because they responded insincerely. Approximately same numbers of female ( n = 128, 
53.1%) and male (n= 113, 46.9%) employees participated in this study. Participants’ age was 20s ( n = 154, 
63.9%), 30s ( n = 60, 24.9%), 40s ( n =11, 4.6%), more than 50s (n = 11, 4.6%). Many participants worked for 
administration ( n = 90, 37.3%), sales ( n = 36. 14.9%), marketing (n = 21, 8.7%), research and development ( n = 
28, 11.6%), and manufacture ( n = 7, 2.9%). Most of their positions were staff ( n = 161, 66.8%), assistant 
manager ( n = 48, 19.9%), manager ( n = 20, 8.3%), deputy general manager ( n = 8, 3.3%), and general manager 
(n = 1, 0.4%) were following. 
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Measures 

In this study, all data were collected using a self-report survey. The survey was conducted from March to April, 
2007. The survey consisted of three sections. The first and second section measured perceived goal structure, 
leadership style, individual goal orientation, self-efficacy, collective efficacy, and intentions of innovative 
behavior. And the last section, participants responded demographic information, including gender, age, job, 
position, years of work at the department and the company. Many participants had worked ‘from 1 year to 5 
years’ ( n = 120, 49.8%) in their company, and ‘less than 1 year’ ( n = 60, 24.9%), ‘from 5 years to 10 years’ ( n = 
44, 18.3%), ‘more than 10 years’ ( n = 12, 5.0%) were following. In addition to that, many participants had 
worked ‘less than 1’ year (n= 111, 46.1%) in their current department, and ‘from 1 year to 5 years’ ( n = 99, 
41.1%), ‘from 5 years to 10 years’ ( n = 19, 7.9%), ‘more than 10 years’ ( n = 3, 3.7%) were following. 

In the first and second section, to avoid central tendency of answer, 6-point scales were used with 1 (strongly 
disagree), 2 (a little disagree), 3 (somewhat disagree), 4 (somewhat agree), 5 (a little agree), 6 (strongly agree). 
They are summarized below. 

Organizational goal structure. These items were taken from Patterns of Adaptive Learning Survey (PALS) 
developed by Midgley and his colleagues (2000). Since these scales were designed to examine classroom goal 
structure, they should be modified to apply corporation. Items for goal structure consisted of learning and 
performance goal structure. Learning goal structure composed of 6 items (a = .79), including “My leader really 
wants us to enjoy learning new things at work”. Performance goal structure was measured with five items (a 
= .67), including “My leader tells us how we are compared with other colleagues”. 

Leadership style. These items are adapted from Tepper and Percy (1994). They abstracted and validated items 
from Bass & Avolio(1990). Transformational leadership consisted of 10 items (a = .92), including “I am ready 
to trust him/her to overcome any obstacle”. Transactional leadership consisted of 6 items (a = .83), including “I 
can get what I want if I work as agreed with him/her”. 

Individual goal orientation. Employees’ learning and performance goal orientations were measured from 
Button, Mathieu, & Zajac (1996). Items for learning goal orientation (a = .81) included “When I fail to complete 
a difficult task, I plan to try harder the next time I work on it. Whereas items for performance goal orientation (a 
= .79) included “I prefer to do things that I can do well rather than things that I do poorly”. 

Self-efficacy. These scales were measured with 7 items adapted from Parker, Williams, & Turner (2006). Items 
included “How confident would you feel helping to set targets in your area”. The reliability was a = .84. 

Collective efficacy. These items were assessed with 9 Items adapted from Lent, Schmidt, & Schmidt (2006). 
Items included “How confident are you that your team could reach agreement about what needs to get done at 
each meeting”. The reliability was a = .87. 

Intentions of Innovative behavior. These items were from Scott and Bruce (1994). Although this scale was 
designed to measure individual innovative behavior, this study applied it to individual and group level 
simultaneously. Items for individual innovative behavior (a = .86) included “I search out new technologies, 
processes, techniques, and/or product ideas”. And Items for group innovative behavior (a = .91) included “Our 
group search out new technologies, processes, techniques, and/or product ideas”. 

Results 



Preliminary Analyses 

Table 1 shows descriptive statistics and zero-order correlations among the variables. First of all, organizational 
learning goal structure closely related to transformational leadership (r = .78), and transactional leadership (r 
= .72). Transformational leadership was also closely related to transactional leadership ( r = .83). They created a 
problem of multicollinearity. Theoretically, learning goal structure and transformational leadership are similar 
because they emphasize challenging the current status and learning new things. However, transactional 
leadership focused on established performance and do not allow exceptional error. So it cannot be theoretically 
combined with learning goal structure and transformational leadership. Therefore, to solve the problem of 
multicollinearity, transactional leadership was removed and learning goal structure and transformational 
leadership were combined with one factor. 

- Table 1 is presented here - 

Besides, to demonstrate the construct validity, exploratory factor analysis using principal axis factoring was 
conducted on learning and performance goal structure, and transformational leadership. Two factors were 
optimal and accounted for 46.57% of the total variance. The six items of learning goal structure and the ten 
items of transformational leadership loaded onto the first factor, and the five items of performance goal structure 
was associated with the second. 



Regression Analyses 
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This study examined the mediating role of individual goal orientations and efficacy among organizational goal 
structure, leadership style, and intentions of innovative behavior. According to Baron and Kenny (1986), to 
demonstrate mediated relations, there must be significant relations between (a) the predictors and the mediators, 
(b) the predictors and the outcomes, and (c) the mediators and the outcomes. Especially, in the third step, all of 
the variables should be entered into the same model and the direct effects of the predictors on the outcomes must 
be reduced. 

For demonstrating mediated effect, this study conducted three sets of regression analyses. In the first set, it is 
assessed the relations between organizational goal structure and leadership style, and individual goal 
orientations. In the second set, it is tested individual goal orientations’ mediating role between organizational 
goal structure and leadership style, and efficacy. In the third set, it is examined whether individual goal 
orientation and efficacy mediated the relations between organizational goal structure and leadership style, and 
intentions of innovative behavior. 

Goal Structure and Leadership Style to Goal Orientations 

As presented in Table 2, organizational goal structure and leadership style predicted a significant amount of 
variance in personal goal orientations, F( 2, 241) = 12.01 ,p< .001. They accounted for 9% of learning goal 
orientation, and 4% of performance goal orientation. Employees who perceived their organization has 
emphasized challenging and learning new things were likely to report that they were willing to put an effort to 
learn new things and improve their ability. On the contrary, employees who viewed organization as emphasizing 
competition and performance reported that it is important for them to demonstrate their ability compared with 
others. 

- Table 2 is presented here - 

Goal Orientations between Goal Structure and Leadership Style, and Efficacy 

To explore mediated effects of goal orientations between goal structure and leadership style, and efficacy, 2 
step hierarchical regression analyses were conducted. The results are presented in Table 3. In the first step, goal 
structure and leadership style accounted for 8% of self-efficacy, F( 2, 241) = 10.15 ,/? < .001; 26% of collective 
efficacy, F( 2, 241) = 42.35, p < .001. It is remarkably seen that a majority of variance for collective efficacy was 
explained by learning goal structure and transformational leadership (p = .52 , p < .001). 

- Table 3 is presented here - 

In the second step, adding individual goal orientations more explained an additional 24% for self-efficacy, F( 4, 
241) = 27.10,/? < .001; 10% for collective efficacy, F(4, 241) = 33.79 ,/? < .001. As expected, learning goal 
orientation strongly predicted self-efficacy (p = .51, p < .001) and collective efficacy (P = .34,/? < .001), even 
when other predictor was considered. In contrast, performance goal orientation did not predict self-efficacy and 
collective efficacy. However, learning goal orientation and transformational leadership stayed a significant 
predictor of self-efficacy (p = .12,/? < .05) and collective efficacy (P = .43,/? < .001). Therefore, personal 
learning goal orientation partially mediated from learning goal structure and transformational leadership to self- 
efficacy and collective efficacy. 

Goal Orientations and Efficacy between Goal Structure and Leadership Style, and Intentions of 
innovative behavior 

Finally, this study testified goal orientations and efficacy’s mediated effects from goal structure and leadership 
style to intentions of innovative behavior. 3 steps of hierarchical regressions for intentions of innovative 
behavior were conducted. First, organizational goal structure and leadership style were entered. As presented 
Table 4, goal structure and leadership style accounted for 15% of intention of individual innovative behavior, 

F( 2, 241) = 21.19,/? < .001, 33% for intention of group innovative behavior, F(2, 241) = 57.58,/? < .001. 
Learning goal structure and transformational leadership positively predicted intention of individual innovative 
behavior (P = .38,/? < .001), and group innovative behavior (p = .58,/? < .001). However, performance goal 
structure did not predict anyone. 

- Table 4 is presented here - 

In the second step, adding individual goal orientation increased the amount of variance explained by additional 
13% for intention of individual innovative behavior, F( 4, 241) = 22.41,/? < .001, additional 3% for intention of 
group innovative behavior, F( 4, 241) = 32.59,/? < .001. When learning goal structure and transformational 
leadership was explained, personal learning goal orientation positively predicted intention of individual 
innovative behavior (P = .37,/? < .001), and also group innovative behavior (P = .18,/? < .001). However, 
personal performance goal orientation did not predict any variables. Interestingly, learning goal structure and 
transformational leadership predicted intention of group innovative behave more strongly than personal learning 
goal orientation (see Table 4). 

In third step, efficacy was added into the equation. They increased the total amount of variance explained by 
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additional 13% for intention of individual innovative behavior, F(6, 241) = 26.69, p < .001, additional 14% for 
intention of group innovative behavior, F(6, 241) = 38.29, p < .001. When learning goal structure and 
transformational leadership, and learning goal orientation were accounted for, self-efficacy strongly predicted 
intention of individual innovative behavior (P = .38,/? < .001). But, it did not predict intention of group 
innovative behavior. On the other hand, collective efficacy strongly predicted intention of group innovative 
behavior (p = .44,/? < .001), whereas it did not predict intention of individual innovative behavior. That is, self- 
efficacy and collective efficacy predicted outcomes discriminately. People who have high self-efficacy were 
willing to take innovative behavior personally, and people who have high collective efficacy were likely to 
behave innovation with other colleagues. 

Consequently, even when organizational goal structure and leadership were accounted for, personal learning 
goal orientation and efficacy positively predicted intentions of innovative behavior. In this sense, employees’ 
work motivation partially mediated between perceived organizational goal structure and leadership style, and 
intentions of innovative behavior. The results are summarized in Figure 1. 

- Figure 1 is presented here - 
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Discussion 



The present study suggested that corporation that is perceived as emphasizing learning new things, challenging 
the status quo, and positively recognizing an error is related to learning goal orientation, self-efficacy, collective 
efficacy, and intentions of innovative behavior. On the other hand, organization that is perceived as emphasizing 
competition, demonstrating performance is not related to these desirable psychological processes. Most of all, 
this results suggested that goal structure theory could apply to corporations. Although goal structure research has 
broadly studied in school or classroom environment, it is hardly ever performed in industries. However, current 
researchers viewed goal orientation as not a trait, but a state that can be easily influenced by situation (Dragoni, 
2005). In this sense, which goal is emphasized in organization is critical to make worker’s goal orientation. 
Theoretical and practical implication is discussed below, as well as limitation of this study. 

Organizational Goal Structure and Leadership Styles 

Although these constructs have never been studied together before, they are theoretically similar. But, this 
study failed to identify this relationship clearly. Learning and performance goal structure classified clearly, 
whereas two contradicting leadership styles were closely associated with each other. One reason of this result is 
that many leaders have used both transactional and transformational leadership to balance the reality with the 
future (Howell & Avolio, 1993) 

Organizational Goal Structure 

This study is consistent with the research findings at schools (Gutman, 2006; Roeser, et al., 1996; Wolters, 
2004). Learning goal structure was closely related to personal learning goal orientation, self and collective 
efficacy, and intentions of innovative behavior. On the other hand, performance goal structure is only related to 
individual performance orientation. This results supported Dragoni (2005) ’s hypotheses. 

Nowadays, companies have to compete with all over the world. Although results are critical for corporations to 
maintain their present, learning should be needed to prepare their future. Further, since the business environment 
has changed very fast, it might be unexpectedly altered. In this situation, the way driven the present results is 
useless to cope with these problems. Therefore, learning and innovation are getting important today’s 
corporations. 

In this sense, the strategy for learning goal structure in classroom or school might be useful in corporation. 
Ames (1992) suggested some strategies for learning goal structure. According to him, it is consisted of task, 
authority, and evaluation/recognition. First, leaders should assign task that is diverse, new, and moderately 
challengeable. And they also support followers to set short-term and self-referenced goal, give some tips to 
achieve these goals. Second, leaders should empower authority to followers. For instance, they are able to let 
followers participate in decision making process, or require responsible and independent attitude at work. And 
also leaders should check follower’s progress. 

Finally, for learning goal structure, leaders should consider how to evaluate or recognize followers’ effort and 
performance. They should praise followers for their improvement and progress. In this process, leaders must 
comment to their followers privately, not publicly. Even if followers make a mistake when they try new things, 
leaders should encourage them and emphasize it could be helpful for better outcomes in the future. 

Work motivation as a mediator 

This study considered a mediating role of work motivation between organizational goal structure and 
leadership styles, and intentions of innovative behavior. First, employees who tend to improve their ability have 
strong self and collective efficacy. And they are willing to take an action for innovation in their workplace. 
However, people who focused on demonstration of ability do not have self and collective efficacy. And, of 
course, they were not prepared to take an action for innovation. It is consistent with previous research findings 
in school or classroom (Bell & Kozlowski, 2002; Porter, 2005; Roeser et al., 1996). 

Second, this study confirmed again that self and collective efficacy had a stronger power to predict outcomes 
than any other variables (Aheame et al., 2005; Locke, Frederick, et al., 1984; Pajares & Miller, 1994; Wood & 
Bandura, 1989). Even when learning goal structure and transformational leadership, and personal learning goal 
orientation were accounted for, self and collective efficacy strongly predicted intentions of innovative behavior. 
Although self-efficacy and collective efficacy was moderately correlated, they are distinguished clearly by 
predicting different outcomes. People who had strong self-efficacy were prepared to challenge the status quo 
individually, whereas those who had strong collective efficacy were willing to innovate collectively. This result 
also supported Wang & Newlin (2002)’s suggestion. 

Limitations 

Several limitations are worth to note for future research. First, the present study only focused on employees’ 
psychological experiences using self-report data. Consequently, it did not explored how people’s perceptions 
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regarding organization related to real policies, systems, or practices in workplaces. They might have a great 
influence on organizational goal structure. Therefore, it is worth to compare objective organizational conditions 
with employees’ perceptions in the future studies. 

Second, this study did not consider participant’s individual characteristics. Perception of goal structure might 
be affected by personal job, gender, or performance level. In the case of adolescence, academic domain, gender, 
ability, and year at school had an effect on perception of classroom goal structure (Anderman & Midgley, 1997). 
In this sense, it should be studied which factors influence perception of organizational goal structure. 

Finally, this study conducted the survey only one time. Hence, it did not consider the effect of performance on 
individual goal orientation, efficacy, and intentions of innovative behavior. Employees’ motivation is able to 
stronger or weaker by their performance. For instance, strong self-efficacy makes people do their tasks 
successfully and also this successful performance increases self-efficacy (Lindsley, Brass, & Thomas, 1995; 
Wood & Bandura, 1989). It is certain that success of innovation might encourage employees to keep learning 
and trying to innovate. That is, there are reciprocal relationships between work motivation and performance over 
time. In this sense, longitudinal measures are needed, including outcomes such as performance in the following 
research. 
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<Figure & Tables> 




Figure 1 

Results of hierarchical regression analyses. N= 241. 
Standardized regression coefficients and R 2 s are presented. 
><.05. *><.001. 

^ Original Figure attached another file. 



Table 1 



Means, Standard Deviations, and Correlations among Variables 



Variable 


M 


SD 


1 


2 


3 4 5 6 


7 8 9 10 


Organizational goal structure 














1 . Learning goal structure 


4.08 


.88 


- 








2. Performance goal structure 
Leadership style 


3.32 


.88 


20 ** 


■ 






3. Transformational leadership 


3.76 


.90 


79 ** 


.26** 


- 




4. Transactional leadership 
Individual goal orientation 


3.62 


.98 


72** 


33** 


83** 




5. Learning goal orientation 


4.56 


.56 


.30** 


.11 


27** 23** 




6 . Performance goal orientation 
Efficacy 


4.35 


.62 


.08 


19** 


15* 21 ** 18** 


- 


7. Self-efficacy 


4.22 


.71 


32** 


.10 


21** 30** 55** 09 


- 


8 . Collective efficacy 
Intentions of innovative behavior 


4.15 


.72 


4 g** 


.08 


.48** .46** .45** .08 


59** 


9. Intention of individual 
innovative behavior 


3.60 


.79 


39 ** 


.11 


34 ** 34 ** 57 ** 49 ** 45 ** 08 


10. Intention of group 
innovative behavior 

** 


3.50 


.93 


.56** 


.12 


53** 45** 41** 54 ** 34 ** 09 59** 



Note. N= 241. > < .05. *> < .001 . 



Table 2 



Regression Analyses Predicting Individual Goal Orientation 



Dependent variable 


Independent variable 


Learning goal 
orientation 


Performance goal 
orientation 


B 


SE B p W~ 


_ B SEB p R i 2- 


Learning goal structure & 
Transformational leadership 


.04 .29** .09** 


.06 .05 .09 .04* 


Performance goal structure .03 


.04 .04 


.12 .05 .17* 



Note. N= 241. > < .05. *> < .001. 
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Table 3 



Hierarchical Regression Analyses Predicting Efficacy 









Dependent variable 


Independent variable 




Self-efficacy 


Collective efficacy 


B 


SEB 


p R z 


B 


SEB P R z 


Step 1 












Learning goal structure & 
Transformational leadership 


.22 


.05 


27** Qg** 


.43 


.05 .52**. 26** 


Performance goal structure 


.03 


.05 


.04 


-.04 


0 

1 

© 


Step 2 

Learning goal structure & 
Transformational leadership 


.10 


.05 


12* 32 ** 


.35 


.05 .43**. 36** 


Performance goal structure 


.01 


.05 


.02 


-.04 


.04 -.06 


Learning goal orientation 
Performance goal orientation 


.64 

-.02 


.07 

.06 


51** 

-.02 


.43 

-.02 


.07 .34** 

.06 -.02 



Note. N = 241. *p < .05. *> < .001. 



Table 4 

Hierarchical Regression Analyses Predicting Intentions of Innovative Behavior 

Dependent variable 



Intention of individual Intention of group 
Independent variable innovative behavior innovative behavior 





B 


SEB 


p R z 


B 


SEB 


p R z 


Step 1 














Learning goal structure & 


.35 


.06 


38** 15** 


.61 


.06 


.58** .33** 


Transformational leadership 














Performance goal structure 


.02 


.06 


.02 


-.02 


.06 


-.02 


Step 2 














Learning goal structure & 


.25 


.05 


.28** .28** 


.55 


.06 


.52** .36** 


Transformational leadership 














Performance goal structure 


.00 


.05 


.00 


-.03 


.06 


-.03 


Learning goal orientation 


.52 


.08 


37 ** 


.30 


.09 


.18* 


Performance goal orientation 


-.03 


.07 


-.02 


-.01 


.08 


-.00 


Step 3 














Learning goal structure & 


.17 


.06 


.18* 41** 


.35 


.06 


.33** .50** 


Transformational leadership 














Performance goal structure 


.00 


.05 


.00 


-.00 


.05 


-.00 


Learning goal orientation 


.20 


.09 


.14* 


.01 


.10 


.01 


Performance goal orientation 


-.02 


.07 


-.01 


.01 


.07 


.01 


Self-efficacy 


.43 


.08 


.38** 


.07 


.08 


.05 


Collective efficacy 


.12 


.08 


.11 


.57 


.08 


44 ** 



Note. N= 241.*p < .05. **p < .001. 
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Assessing the Use of Video Conferencing for Linking 
Teacher Preparation Programs with K-12 Schools 

James D. Lehman and Jennifer Richardson, Purdue University 

Abstract 

Teacher preparation programs face many challenges including preparing future teachers to work in 
diverse classrooms and to effectively use technology. Video conferencing technology offers a means for 
teacher preparation programs to broaden opportunities for future teachers to obtain desired knowledge and 
skills through linkages with K-12 schools at a distance from campus. Projects at Purdue University have 
explored the potential of linking teacher education classes/candidates with K-12 teachers/students via video 
conferencing. Data collected from participating university and school faculty as well as from participating pre- 
service teachers suggest this approach has certain benefits but also limitations. Project results suggest that this 
approach may be a useful way to augment traditional field experiences in teacher preparation programs. 



Introduction 

University teacher preparation programs face many challenges today. Over the past two decades, a 
number of national reports have emphasized the need to improve teacher preparation (Carnegie Forum, 1986; 
Holmes Group, 1986; Levine, 2006; National Commission on Teaching and America's Future, 1996). Today, 
teacher preparation programs must prepare future teachers to meet national and state standards with regard to 
both content and pedagogy in a time of increased emphasis on performance and accountability. Colleges of 
education must also help pre-service teachers learn to effectively use technology (ISTE, 2002) and develop 
their understanding of diversity and multiculturalism (NCATE, 2001). Meeting these various standards and 
requirements is difficult in the best of circumstances, and it may require college of education to find new 
approaches to prepare prospective teachers for the classrooms of today and tomorrow. 

Field experiences are a key means to better prepare teachers for the diversity and complexity of today's 
classrooms (Goodlad, 1990). While field experiences are generally recognized as critically important and rated 
as the most valuable aspect of preparation by prospective teachers, the amount of field experience in teacher 
preparation programs is often limited (Levine, 2006). Further, many colleges of education, particularly those in 
rural areas, have difficulty placing students in field settings that provide for needed experiences with, for 
example, diverse student populations. New technologies, such as Internet-based video conferencing, offer 
capabilities that might be employed to provide needed experiences for pre-service teachers when appropriate 
field sites are not in close proximity. To explore this potential, Purdue University initiated an innovative 
project in the use of technology-enabled field experiences as part of a Preparing Tomorrow's Teachers to use 
Technology (PT3) implementation grant (Lehman, 2003; Lehman, Richardson, Malewski, & Phillion, 2005). 
This project was designed to address key components of Purdue’s teacher preparation program, including 
understanding of classrooms and diversity as well as appropriate technology integration, through video 
conferencing connections with K-12 schools at a distance. 

Video conferencing technologies allow teacher educators and future teachers to connect in real time 
with distant K-12 classrooms to observe and interact with students and teachers. This can provide 
opportunities for future teachers to work with schools and student populations that might otherwise be 
unavailable for traditional field experiences. Further, the process of using this cutting-edge technology 
simultaneously provides a model of technology integration for pre-service teachers. In Purdue’s project, 
different models or ways of using the technology for remote observations and virtual field experiences were 
explored, including short-term connections focused primarily on pre-service teacher observations of 
classrooms, longer-term connections in which pre-service teachers actively worked with school children, and 
distant supervision of student teachers by teacher education faculty members. The objective of this study was 
to assess participants’ perceptions of the benefits and limitations of this use of video conferencing as a tool for 
linking the university with K-12 school sites at a distance. Participating pre-service teachers were surveyed, 
and participating faculty members and classroom teachers from various pilot projects were interviewed. The 
data, both quantitative and qualitative, were analyzed to identify perceived benefits and limitations of this 
approach. 
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Background Literature 

The concept of using live video to allow pre-service teachers to observe K-12 classrooms has been 
around for a number of years. Closed circuit television was used for observation of classrooms in teacher 
education programs as far back as 1960 (c.f. Abel, 1960). In the 1980s, Iowa State University's Teachers on 
Television program showed that the observation skills of pre-service elementary teachers could be improved 
through practice that involved viewing live microwave-based video broadcasts from public school classrooms 
(Hoy & Merkley, 1989). While providing proof of concept, early applications of closed circuit and microwave 
video for observations of K-12 classrooms were expensive, difficult to set up and maintain, and typically 
provided only one-way video from the K-12 classroom into the university classroom. 

Today’s video conferencing technologies are a much more flexible, capable, and cost-effective option 
for observation of and interaction with school-aged learners at remote school sites. Modestly priced equipment 
for video conferencing over the Internet, so-called IP-based video conferencing, is now available. Using this 
equipment, a two-way audio and video connection can be established between any two sites, such as university 
classroom and a K-12 classroom or other instructional space, equipped with a local area network and relatively 
high speed Internet connection. Thus, video conferencing today has great potential for helping universities and 
schools to make real-time connections. 

Recent literature concerning applications of video conferencing in education, particularly involving 
the linking of universities and K-12 schools, is limited. Uses of video conferencing that have been reported 
include virtual field trips, remote observation of teaching practicum students, teacher professional development, 
and what might be termed virtual field experiences for prospective teachers. 

Virtual field trips involve the use of video conferencing for short-term experiences where K-12 or 
college students connect to a distant site to learn more about the site or engage in a planned activity (LeBlanc, 
2002; Pachnowski, 2002). In Indiana, for example, K-12 schools that are part of an in-state video network are 
able to connect to informal education service providers such as the Indianapolis Zoo and the Indianapolis 
Children’s Museum through two-way video conferencing to learn about exhibits and interact with educational 
personnel. Such uses of video conferencing allow students to experience resources at remote locations without 
having to travel to the locations. 

Some universities have experimented with the use of video conferencing as a tool for supervising 
practicum students at a distance (Pemberton, Tyler- Wood, Perez Cereijo, Rademacher, & Mortensen, 2001; 
Sharpe, Hu, Crawford, Gopinathan, Moo, & Wong, 2000) and for conducting teacher professional development 
(McDevitt, 1996; Welch & Sheridan, 2000). Although it was not initially an objective of our project at Purdue, 
several faculty members experimented with the use of video conferencing for observing practicum students, i.e. 
student teachers, at a distance. Initial pilot projects led to an internal grant from the university that funded the 
acquisition of additional equipment to support faculty members wishing to use the technology for remote 
supervision and other connections to schools. 

Video conferencing can also support a form of virtual field experience in which pre-service teachers 
can not only observe classrooms but, thanks to the two-way audio and video, also interact with students and 
teachers at the school site. For example, McDevitt (1996) described the use of two-way video conferencing to 
allow pre-service teachers to observe professional development school sites and interact with teachers about the 
observed lessons. This approach gave pre-service teachers the opportunity to observe expert teachers in action 
and talk with them about their teaching, and it also aroused the pre-service teachers’ interest in using the 
technology themselves. Edens (2001) discussed the advantages and limitations of using videoconferencing to 
link a university teacher education class to first-grade and fifth-grade classrooms for, among other things, 
question and answer exchanges between the pre-service teachers and the elementary students. Howland and 
Wedman (2003) discussed the use of videoconferencing as a way to help pre-service teachers experience 
various forms of diversity, including cross-cultural learning through video conferencing connections with 
overseas schools. Most of the initial efforts in our project explored similar uses of video conferencing 
technology to provide opportunities for teacher education candidates to observe and interact with K-12 
students and teachers in partner school districts that included diverse, urban schools unlike those near to the 
campus (Lehman, Richardson, Malewski, & Phillion, 2005; O’Connor, 2003; Phillion, Johnson, & Lehman, 
2003-04). 
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Description of the Project 

This project was one part of a Preparing Tomorrow’s Teachers to use Technology (PT3) 
implementation grant conducted from 2000 to 2004. The overarching goals of Purdue’s PT3 grant were to (1) 
prepare pre-service teachers to demonstrate fundamental technology competencies, using technology as a tool 
for teaching/learning, personal productivity, communication, and reflection on their teaching, and (2) prepare 
teacher education faculty to teach pre-service teachers in technology-rich environments, modeling approaches 
that future teachers should use themselves. The project addressed its goals via several complementary 
components including the creation of technology-enabled field experiences through video conferencing 
linkages with partner K-12 schools. 

For colleges of education in rural locations, like Purdue, it can be difficult to place students in field 
settings that provide important experiences such as interaction with diverse student populations. As one way to 
address this problem, the PT3 project made use of two-way video conferencing to link pre-service teachers and 
teacher education classes with K-12 students and teachers at distant school locations that included urban 
centers with diverse student populations. This pilot initiative was designed to explore models for enhancing 
teacher preparation through these linkages between the university and participating K-12 schools. 

Two types of video conferencing were used to enable diverse field experiences for students using 
technology. At the outset of the project, we planned to use an intrastate fiber optic video network called 
Vision Athena, managed by the Center for Interactive Learning and Collaboration ( http://www.cilc.org ), a 
partner in our project. While we used that network on a limited basis to link to some of our partner schools, 
IP-based video conferencing equipment from Polycom ( http://www.polycom.com ) emerged during the early 
stages of the project as a better way to meet most of our needs. This technology supports good quality video 
and audio over the Internet, is relatively affordable, and is very flexible because a standard H.323 Internet 
video conferencing connection can be established between any two locations with access to a reasonably fast 
(128 Kbps or better) connection. IP-based video conferencing does not require special distance education 
rooms or video studios; connections can be established from classroom to classroom. 

We used two main types of Polycom video conferencing equipment. Room-to-room video 
conferencing was supported by equipment such as the Viewstation SP (point-to-point) or Viewstation FX 
(multipoint) unit. (Today, Polycom’s VSX series provides this capability.) These units have an integrated 
camera with panning and zooming capability that can be controlled remotely. One can be attached to any 
available video monitor and plugged into an Ethernet jack for Internet connectivity. Educational prices start at 
about $2,500 for a point-to-point unit capable of handling a classroom; two units are needed, one on each end 
of the connection. For person-to-person or small-group-to-small-group connectivity, we used the Polycom 
ViaVideo computer-based desktop video conferencing unit connected to a Windows PC. (Today’s model is the 
Polycom PVX.) While the smaller desktop video camera is of lesser quality and lacks the panning and 
zooming capability of the larger Viewstation units, this inexpensive (about $400) unit provides basic person-to- 
person connectivity while adding the ability to share computer applications during video conferencing. For 
basic observations by one or two individuals, a desktop unit can be used on one end of the connection linking 
to a room-based unit on the other end. While the costs of these video conferencing units are not trivial, they are 
substantially less than video conferencing costs of just a decade or so ago, and the costs must be weighed 
against the savings in travel time and expense that can be realized from the use of video conferencing. 

With the assistance of PT3 project staff, faculty members interested in using the equipment to connect 
with schools made contacts with teachers in the partner schools. Tentative plans for some sort of video 
connection were made between the faculty member and the participating K-12 teacher in consultation with the 
PT3 project staff, school officials, and technical support staff. The project provided equipment to support the 
activity to the remote school site. Technical support staff in Purdue’s College of Education then worked with 
the technical support staff in the school to set up, test, and maintain the video linkage between the university 
and the school. The chief obstacle in establishing the connection was creating the necessary point of entry 
through the school’s Internet firewall, which otherwise would block the use of the equipment. This typically 
required some reconfiguration of the school’s firewall to allow the Polycom equipment to function properly. 

Several ways of using the video conferencing connections to link teacher education candidates and 
faculty with K-12 classrooms were explored in our project. Although there were a number of variations, three 
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main types of interactions emerged: (1) short-term class activities, typically involving only one or two video 
connections, that focused mainly on having pre-service teachers observe a classroom in action and perhaps 
interact with the teacher about the lesson; (2) longer-term class activities, with regular video connections 
stretching over a period of weeks, in which pre-service teachers had the opportunity to interact with and 
actually teach lessons to the students in the remote classroom; and (3) remote observations of student teachers 
in the classroom by university faculty members. 

In one example of a short-term class experience, a faculty member in consumer and family science 
education had her class of pre-service teachers spend a class period observing a pre-school, housed in a high 
school in an urban center in the state, to learn more about its operation. The pre-service teachers were able to 
observe the center and interact with the teacher about its operation. In another short-term experience, a faculty 
member in mathematics education had her mathematics education college students spend two class periods 
observing mathematics problem-solving activities in an elementary school classroom. Using the ability to 
control the camera in the classroom, the college students were able to zoom in on selected groups of elementary 
students to observe how they interacted with one another during the problem-solving activities. 

The longest running pilot project involved multiple interactions between a class of pre-service teachers 
and a diverse elementary school classroom in an inner city Indiana community. Guided by the faculty member 
in charge of the course, a section of pre-service teachers in their first teacher education course connected to the 
elementary classroom once per week for about one to two hours throughout most of a semester. This activity 
took the place of a traditional early field experience in which the pre-service teachers would have been 
assigned to various classrooms for two hours of observations per week. In the video conferencing experience, 
the pre-service teachers initially observed the classroom remotely but gradually expanded their efforts to 
eventually teach lessons to the school children at a distance to complement the teacher’s curriculum. This 
project was described in a video program produced for WHRO-TV by Soundprint Media for the Teaching 
Now! video series (available online at http ://teachingnow. org/watchT V.php?id=3 0 j . The interaction between a 
section of the teacher education course and the elementary school was repeated over the course of several 
semesters. 

Remote observation of practicum students, i.e. student teachers, was added in the final year of the 
project. Faculty members in both science education and consumer and family sciences education experimented 
with remote observation of student teachers as a substitute for some but not all of the normal in-person 
observations. The video conferencing allowed the faculty supervisors to unobtrusively observe student teachers 
in the classroom without having to travel to the school site. 

Each of these pilot uses of video conferencing offered potential benefits, but also had drawbacks. The 
intent of this study was to examine participants’ perceptions of the benefits and limitations of these uses of 
video conferencing for observations and virtual field experiences. Data from pre-service teachers, university 
faculty members, and classroom teachers were analyzed to identify perceived benefits and limitations of this 
approach. 



Methods and Data Sources 

Two primary sources of data were utilized in this study. First, pre-service teachers who participated in 
the long-term video conferencing experience, in which an introductory teacher education class connected to an 
elementary classroom once per week throughout most of a semester, were surveyed following their 
experiences. These surveys were administered to two separate groups of pre-service teacher participants (n=21 
each) in each of two semesters of the project during an academic year. Surveys consisted of Likert-like and 
open-ended items designed to elicit candidates’ perceptions of the video conferencing field experiences. 
Surveys were administered online, and were completed by students after they had participated in multiple video 
conferencing experiences with participating K-12 students and teachers. Data were compiled, including mean 
responses to Likert-like items and common responses to open-ended items. 

Second, to gather more in-depth qualitative data, interviews were conducted near the end of the 
project with seven faculty members and five classroom teachers who participated in the video conferencing 
experiences and analyzed using a qualitative-interpretative research paradigm. Semi-structured interviews 
were employed to gather information about the nature of the video conferencing activities and participants’ 
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perceptions of them. Interviews were transcribed and constant comparative analysis (Strauss & Corbin, 1990) 
was employed to identify common themes related to benefits and limitations of the video conferencing 
experiences. 

Results 



Pre-service Teachers 

Pre-service teachers’ responses to the Likert- type survey items about the video conferencing 
experiences are summarized in Table 1 . These data suggest that the pre-service teachers who participated in this 
experience had relatively positive responses to it; mean responses clustered near a response of ’’agree.” The pre- 
service teachers tended to agree that they were comfortable with the technology, found it easy to use, learned 
how to use it as a result of the experiences, and found it to be of value to the class. They also tended to agree 
that the experience made them more comfortable in their ability to use technology for teaching and learning and 
in their ability to understand and work with diverse learners. Although there were some minor differences, the 
pattern of responses across the two different groups was similar. 



Table 1 

Summary of Pre-Service Teachers’ Responses to Likert-type Survey Items 

Likert-type Survey Item Group 1 Group 2 

Mean Mean 

(n=2 1 ) (n=21) 

By the end of the class, I felt comfortable with video conferencing 3.90 3.71 

equipment that we used. 

The video conferencing in this class was easy to use. 4.00 4.00 

I learned how to use video conferencing in education from this class. 3.95 3.67 

I believe that the use of video conferencing was a valuable addition 3.95 3.33 

to this class. 

Because of the experience in this class, I feel more comfortable in 3.90 3.62 

my ability to use technology for teaching and learning. 

Because of the experience in this class, I feel more comfortable in 3.90 4.14 

my ability to understand and teach diverse learners. 



Note: means on scale of 5 = strongly agree, 4 = agree, 3 = undecided, 2 = disagree, 1 = strongly disagree 



The pre-service teachers’ comments in response to open-ended survey items provided further insight 
into their perceptions of the experience. One of the key advantages cited by the pre-service teachers was the 
ability to observe a classroom at a distance. One pre-service teacher noted, "We got to watch the students more 
in their element (because they couldn’t see us, so they did not act as different).” Another commented, "I think it 
was nice to be able to watch the students without being right in the classroom. We were able to see them at the 
end with them practically forgetting we were there.” Another pre-service teacher liked the approach because, 
“We did not have to all be placed in different schools, and we did not have to leave campus on a weekly basis 
to go to a local school. We got a chance to see a classroom without actually being there." 

A second advantage cited by the pre-service teachers was learning to use the technology itself. One 
pre-service teacher reported, ”1 have learned how to use video conferencing to some degree.” Another wrote, 

“I think that I am comfortable with using technology in the classroom.” Another future teacher summed up the 
advantage nicely by saying, “Not only did we get to observe a classroom but we also got to learn how to use 
new technology to do this. I think it was very good to learn this because it most likely will be beneficial for us 
in the future.” 

A third advantage cited by the pre-service teachers related to their exposure to diversity. As one pre- 
service teacher reported, “I think the diversity of the class that we taught was the biggest advantage.” Another 
pointed to, “The opportunity to get to experience a multicultural diverse classroom was a unique experience 
that fit nicely with our block one coursework. It was interesting to view how different their world is compared 
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to the world in which we grew up in.” A third commented that it was an advantage, “Being able to see a very 
diverse classroom and how that is handled by an experienced teacher.” 

The limitations cited by the pre- service teachers fell into two basic areas, technical 
problems/limitations and not being present in the classroom. One pre-service teacher commented, “Sometimes 
the connection could not be completed and when it was at times it was blurry and slow.” Another noted, 
“There were definitely disadvantages to this too. We could never fully interact with the students because you 
couldn’t see them, hear them.” A third remarked, “At times the technology didn’t work and was a distraction to 
the students.” The issue of the lack of personal presence was noted by a Purdue student who wrote, “I felt that 
it was a disadvantage that we were not in the classroom in person.” Another put the issue succinctly saying, 
“There was not a personal relationship built up between us and the students because there were TVs between 
us.” 



University Faculty 

Through our interviews with university faculty members who participated in these video conferencing 
linkages with the schools, we explored their perceptions of benefits and limitations as well. Several themes 
emerged from the analysis of these interviews, and many of these themes mirrored the perceptions of the 
teacher education students. 

One of the themes that emerged from the faculty interviews as an advantage of this approach was the 
ability for students to see a classroom and an experienced teacher in action. Professor Keith (a pseudonym), 
who had her English education students observe a high school classroom, saw an advantage for her students by 
Allowing them to see that real world connection. . . Seeing an experienced teacher actually 
doing some of the things that we were talking about in class as being good things to do, I 
think lends those strategies the credibility that sometimes, if were talking about something 
in class and didn’t see that actually in the high school class, that they might not certainly 
believe that it was a good thing to do. 

Professor Bates, who had her mathematics education students observe elementary school students doing 
problem solving, commented that it is 

very very hard, almost impossible, to find elementary school classes that teach 
mathematics as the national curriculum reform would like... So it’s a chance for them to 
see a real teacher and believe it could happen and then see how he/she conducted the 
class. Very important, they don’t have many models that they have a chance to see what 
this might look like. . . And this is an excellent example actually too, not just of a classroom 
like that, but someone who is really really good at teaching. That, I think, was a big benefit. 

Professor Jones, who integrated observations into her beginning teacher education course, commented on 
the insights the experience provided for her about her teacher education students and their observational 
skills, saying 

Through sitting in one room with these students and all observing the same teacher teach, 
and debriefing and having discussions around that, I realized that they didn’t see the same 
things at all that I see. Because as an experienced teacher you learn to view, for example, 
student-teacher interaction in a different way than someone who’s seeing it from a student 
perspective. 

Like the pre-service teachers, the faculty members also saw benefits in exposing the university 
students to the relatively new technology of video conferencing. Professor McDonald, who required his 
students in a beginning teacher education course to observe a middle school classroom, identified learning 
about technology as one of his goals, saying, “I wanted to get my students to experience the technology, and 
get them to think critically about how they would use it in their programs as a teacher.” This exposure to 
technology benefited not just the students but some of the faculty members themselves. Professor Jones, 
who also used the video conferencing in a beginning teacher education course, commented, 

Pm not into technology. I don’t use much technology. So one of the objectives was to bring 
myself a little more into the world of technology. It’s also part of the mandate for [the 
course] that students will be exposed to exemplary uses of technology, and that students 
will see or model use of technology, and that they’ll learn about technology that they could 
use with their students. So that’s one of the reasons I wanted to do it. 
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Use of the equipment itself afforded certain advantages. Professor Grant used the technology to 
observe student teachers at a distance. She commented 

It’s really kind of neat, because you could adjust the camera to scan the whole room, 
actually zoom right in to what the high school students were preparing. Very different in 
the sense that I could zoom in, see what's going on, and not be distracting to them. 

Whereas if I was there in person, I would've had to get up and walk away from where I 
was sitting, to see what they are doing. 

She also commented on her own use of the technology, noting 

The equipment was really much easier to use, that's a part of my skepticism because I was 
not familiar with it. But you know, really a low level of background, I had a low level of 
background, and it didn't matter. I was able to learn what I needed very easily. 

Professor Hawkins also used the technology for remote student teacher supervision. He reported 

I feel the most beneficial was the ability to make extra visits without driving over there, 
and I think to observe in another way was beneficial. I also liked the fact that I could, you 
know, I would dial in before school started, we could talk, or if the kids were there for 
special, I could talk to the teacher. 

Exposure to diversity was another important theme identified by several of the faculty members. 
Professor Evans used video conferencing in the context of a teacher education class on multiculturalism in 
education. He remarked, “Purdue students do not have much exposure to the types of diversity that I value. 
That will be race, class, gender types of differences. I found it beneficial they connected with students, to 
have some exposures to those issues.” He went on to observe 

They began to see how the reading, the theoretical stuff they are doing in multicultural 
education, applies to school. They see a curriculum of a teacher working in a 
multicultural, multilingual school and multilingual classroom. They can actually see the 
things they read about laid out in the teacher's classroom. 

Professor Jones, whose beginning teacher education course shared a field experience with the 
multiculturalism course, found that connecting to a classroom in an urban, diverse school gave her teacher 
education students some much-needed perspective on the students in such schools. She explained 
The opportunity to have my students interact with children different than children they had 
grown up with was something that I decided would be very valuable for them. The 
opportunity to work with teachers who are committed to working with these kids. These 
teachers are absolutely awesome teachers who work with all sorts of children (ESL, 
bilingual, low SES), and they treat them like they're not deficient. My students think they 
come from single parent or poor families, whatever, and are going to have these deficits. 

And actually they think that this would impede their learning. They start to observe a 
classroom like that, and teach these students over the Internet, and start to realize that these 
children are not deficient; they're bright. One of my students said to me, "I don't know how 
am I going to be able to teach kids like this; they're just so smart." 

The faculty members also saw benefits of the project for other participants. Professor Grant found the 
teachers she worked with to be open to this new approach, saying “All three teachers were very willing to try 
something new, even though they had not done this type of videoconferencing. They were all aware and 
knew it was out there.” Professor Jones saw benefits for the teacher she worked with, noting, “There are 
benefits for the teachers too. [The teacher I worked with] is using this for National Board Certification, she 
won a Crystal Apple Award here, she won best teacher. . . I'm not saying it's just this that just did that, but 
it's part of a recognition that she's a really great teacher who's trying things.” Professor Jones also felt that 
the school children benefited from their interaction with Purdue students. She explained, 

I think it's good for the kids. . . They don't have maybe anyone in their family who has 
gone to college or even talked about college. And suddenly they're talking to college kids 
on a weekly basis, and they say things like "What do you guys eat?" My students will say 
"We eat pizza and hamburgers." "Oh, you can eat pizza and hamburgers at college?"... I 
don't know what they think a university is, but it's somewhat demystified. Maybe I'm 
Pollyannish, but I think that all those connections count. I remember one student saying 
that no one in her family had gone to college, and one day she came on a field trip to 
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Purdue, and then that changed her whole life. She realized that university wasn’t what she 
thought. I’m hoping that some of those kids can think that maybe I can go to college too. 

And they just love, they call them, their Purdue friends, buddies. They love just being 
connected to university students. 

Although the faculty members saw a number of clear benefits of the video conferencing linkages, they 
also recognized problems and limitations. The technology, while generally working fairly well, did have 
problems. Professor Keith noted, “The control of the camera was a little awkward.” She also observed, “I 
think [the pre-service teachers] found the quality of the video interaction a little bit more bothersome than 
what I did. I don’t think they were sure of what the real purpose of it was to be honest.” Professor Hawkins 
commented on the poor connection to one site, saying, “It was disappointing at the elementary because, in 
many instances, the teacher had to unplug their email. There is only one plug, so they had to unplug their 
computer, plug in the Polycom. It was slow, and it wasn’t a good picture. Sometimes, you couldn’t tell who 
was the teacher, who was student teacher.” Professor Jones said, “I thought the technology would be better 
than what it was. There were quite a few problems.” She continued, “The imperfection of the technology, 
and the fact that they’re not in a real classroom is a problem. You’re looking at the kids in the Internet, on a 
TV screen. That is not the same as being right there where you can see every nuance of their facial 
expression.” 

Nearly all of the faculty members cited logistical problems and the work involved in conducting 
virtual sessions as a limitation. Professor Keith observed, “It was really hard to schedule because of time.” 
Professor McDonald echoed this concern, saying, “Schedules create a big barrier.” Professor Grant said, 

I think another surprising aspect to me is that it does take a lot of coordination, a lot 
practices, a lot layers when I started creating it, all those pieces facilitating success. There 
really is team effort. I’m not sure that surprised me, but it certainly, it really like does take 
several people working together to make this happen. 

Professor Jones commented, “I think it is too labor intensive to do this kind of project. It takes a huge 
amount of time.” All agreed that technology support was crucial for success. Professor Grant said, “The 
key factor... is good technical support.” Professor McDonald stated, “If I hadn’t had the technical support, I 
would have been discouraged from doing it. The technical support are really people who deserve the 
congratulations for a job well done. It wouldn’t have happened without them.” 

Despite the limitations, the overall sense among most of the faculty participants was fairly positive. 
Professor Keith said, “I really don’t think there was anything bad about it. It was just I needed to work more 
in terms of my curriculum. . . If I would have to do it again I’d like to more fully integrate it into my course 
plan.” Professor Evans commented, “The response overall is supportive. They really like the experience of 
connecting with [the school site].” Professor Jones captured these mixed sentiments, noting 
At the end [the pre- service teachers] would be fairly happy with what they’ve 
accomplished and what they’ve done, but they also indicate some regret around not having 
been in a regular classroom. . . So it’s a continuum of responses from “I don’t like this 
technology thing, I wish I’d been in a real classroom” to “This was fantastic, I’d like to try 
this in my class” and kind of anything between the two depending on the students. Overall, 
it had to be positive enough for me to continue doing this. 

K-12 Teachers 

A sample of the participating K-12 teachers was also interviewed about the benefits and limitations of 
the video conferencing approach. As a group, they had less to say than the pre-service teachers and university 
faculty members. However, their comments echo many of the themes reported previously. 

The teachers were generally positive about their experiences. One teacher remarked, “Me and the 
class all enjoyed it. Overall it was a very positive... The questioning and answering was good.” Another 
teacher reported, “It was very positive experience. Had virtual tour. Did question and answer... It was 
fascinating process to use technology in that way. I think it’s a positive experience for us all.” She 
continued on to say, “First of all, I guess I was a little skeptical about how the technology would work. The 
technology worked more successfully than I thought. The college students were well-prepared and had 
questions. It worked better than I thought.” 
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Several comments from the teachers supported the diversity theme mentioned by both the pre-service 
teachers and university faculty members. One of the teachers commented 

I think it’s a valuable tool for multicultural classes. Instead of just talking about it in class, 
they can go where these children live. Students should go to culturally diverse classes so that 
they can become more sympathetic and understanding of these children. 

Another expressed similar feeling by saying 

I teach in a culturally diverse school. By connecting to Purdue, the college students can see 
different things than they can in the schools of Lafayette area. We celebrate our diversity. We 
see it as our strength. It's an advantage to use media to see such diversity. College students 
can gain some insights of the diverse classroom. 

The limitations cited by the teachers included the technical problems cited by other groups. One 
teacher commented, “Some technical glitches were a bit frustrating, but not much.” Another noted that 
“When the screen was pixelating, or with the whole group setting, the attention did waver.” A few of the 
teachers expressed initial discomfort at having their classrooms opened to observations. One of the teachers 
explained, “At first I was a bit nervous, because I didn't want to be judged. I've been teaching 20 years, and 
it feels a bit uncomfortable when suddenly many students are watching you.” Finally, for some of the 
teachers, there was little benefit for participating in the experience. One teacher said, “I think it was 
beneficial for the Purdue students, but not really for us.” Another echoed this comment, saying, “Happy to 
do it. But no particular advantages for my class to do that.” 



Conclusions and Implications 

This study examined the benefits and limitations of the use of video conferencing as a tool for linking 
teacher educators and pre-service teachers with K-12 teachers and students at a distance. Different models for 
making connections between one university’s teacher education program and participating K-12 schools were 
explored. Pre-service teachers, university faculty members, and participating K-12 teachers shared their 
perceptions of this innovative project. 

There was agreement across the groups of participants that this use of video conferencing offered 
certain benefits. Among the most compelling benefit was the ability to expose pre-service teachers to examples 
of classroom diversity that might otherwise be difficult or impossible for them to experience. University 
faculty, classroom teachers, and pre-service teachers themselves all agreed that the ability to observe and 
interact with diverse classrooms was a positive outcome of the project. For colleges of education located in 
rural areas, such as Purdue, video conferencing offers one means to allow pre-service teachers, who themselves 
are often from rural and homogenous communities, to have experience with and so come to better understand 
the diversity that increasingly characterizes America’s classrooms. This understanding is a first step toward 
learning how to teach in culturally responsive ways (Gay, 2000; Ladson-Billings, 2001). 

These video conferencing linkages also provided pre-service teachers with examples of classroom 
practice that they might not otherwise get a chance to observe. Theyhad the opportunity to observe expert 
teachers in action, and they had the opportunity to observe specific kinds of classrooms and classroom activities 
(e.g., diverse and multilingual classroom, reform-oriented mathematics problem-solving, in-school child care 
facility) that often are unavailable near the university. While recorded video of such settings might suffice in 
some circumstances, the live video connection offered interactivity and hence a dynamism that recorded video 
lacks. As Professor Jones commented in her interview, “Some people say ‘Why not just tape the teacher 
and show it to everyone?’. That's not it. Passively watching a tape is not the point. It's to be part of the 
experience... Without that interactive element, it would be something I wouldn't have used at all.” 

Of course, this approach also helped the pre-service teachers, as well as the faculty and teachers, 
learn about and use a cutting-edge technology. Technology integration and use, after all, was the central 
focus of the PT3 initiative in which this project took place. Through the project, the pre-service teachers 
began to see the technology as a tool that could be used for teaching and their own and others’ learning. As the 
cost of video conferencing equipment continues to decline, and as network capacity expands, it is likely that 
this technology will become much more widespread in the future. Thus, by exposing prospective teachers to 
the technology today, we plant the seeds for what may be more and more effective use in the schools of the 
future. 
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Despite these benefits, there are clear limitations that must be weighed when considering any potential 
implementation of this approach. The technology itself, while pretty good, is far from perfect. All three groups 
of participants cited technical difficulties that ranged from a complete failure to connect on some occasions to 
pixelation (video break-up) and slowed transmission to the inability to clearly see and hear participants on the 
far end of the connection. These problems can limit, sometimes severely, the utility of video conferencing as a 
tool for real-time communication between universities and schools. However, as Professor Bates observed, 
“The equipment is awkward. The technology is awkward. But, it's going to get better.” We can expect the 
technical problems to diminish as the technology itself matures. 

Other limitations, however, are likely to continue to be issues. For the pre-service teachers, the video 
conferencing experiences lacked some of the authenticity of an actual field experience. The physical distance also 
created some psychological distance, such that it was difficult for the university students to create relationships 
with the K-12 students. As the one pre-service teacher so aptly put it, “there were TVs between us.” 

For the university faculty members and school teachers, the time and effort involved in working out the 
logistics of making video conferencing connections was an issue. Schedules had to be aligned, class activities 
planned, and communication techniques practiced. This added a significant extra burden to their preparations, and 
this occurred despite the fact that during the project PT3 and IT staff members assisted in working out the logistics 
of the connections. Without this added support, it would be very difficult for faculty members and teachers to 
work out video conferencing connections on their own. 

Finally, an important issue for the teachers and schools is the benefit of being involved in this sort of 
activity. Two of the teachers interviewed after the project felt that the project was worthwhile for the teacher 
education students but not particularly of value to themselves or their students. This may not have been true in 
every case. Professor Jones reported that she felt there were benefits both for the teacher and for the K-12 school 
children with whom her class worked. However, it probably is true that these experiences are of more benefit to 
the pre-service teachers than to the other participants. Therefore, universities must consider how best to craft such 
experiences so that mutual benefits accrue to the K-12 teachers and students. 

This study suggests that technology-enabled field experiences may be a viable option for some types 
of student observations and for interactions between teacher education candidates and K-12 students and 
teachers. Video conferencing over the Internet is a new tool that expands the options available to teacher 
preparation programs to help future teachers develop their knowledge and skills. If the limitations of this 
approach are acknowledged and appropriate steps are taken to minimize the shortcomings, this approach may 
give teacher education program more flexibility in addressing the field experience needs of their pre-service 
teachers. 



When we consider all factors these virtual field experiences seem to be a worthwhile way to expose 
pre-service teachers to experiences they might not otherwise get. Our teacher education program has at its core 
emphases on early and continued field experiences, on developing technological skills, and on understanding 
diverse learners. Virtual field experiences through video conferencing offer promise as a way to expand the 
options for linking pre-service teachers with K-12 teachers and students. While we do not advocate replacing 
traditional field experiences with virtual field experiences, these experiences do offer a way to augment the 
experiences of prospective teachers in university teacher preparation programs. 
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The Current Status and Future Prospects 
of Corporate e-Learning in Korea 

Cheolil Lim, Hyeonmi Hong, Eunkyoung Yeon 
Seoul National University 

The corporate e-learning in Korea has grown rapidly over the previous six years (2000-2005). This study 
argues that the main cause of this heightened interest in corporate e-leaming in Korea was not that companies 
needed to provide high-quality training programs through the Internet but that the government took initiative to 
transform the state into an information-based society. The policies for quantitative growth with minimum levels of 
quality and uniformity have been dominant and have resulted in the lack of diverse e-leaming types for authentic 
practices in workplaces. This paper suggests that in order to cope with the new competency requirements of 
employees, corporate e-leaming should be guided both by governmental support and by company initiative. 

Keywords: Corporate e-Leaming, Korea 



I. Introduction 

Open and distance learning in Korea had not been fully implemented and discussed prior to 2000. Until 
then, since 1972, systematic formal education for adult learners who did not have the opportunities to enter more 
traditional colleges had only been provided at such a mega university as Korean National Open University (KNOU). 
Traditional, face-to-face education had been the dominant mode of teaching, while distance learning made up only a 
small portion of the Korean educational system. However, nine cyber universities, which were established in 2001, 
and the new Internet correspondence training policy for corporate e-learning by the Ministry of Labor in 1999 
initiated profound changes in the Korean educational system (Lim, 2003). Now many Korean adult learners can 
pursue an education through the various learning technologies of distance learning. Online college courses are 
delivered via printed material, radio, television, MP3, PMP (portable multimedia player), and the Internet. Also, at 
present, distance corporate training programs are delivered by mail and online as e-learning. 

The unprecedented growth of corporate e-leaming in Korea has been a major feature of distance learning 
since 2000. The Ministry of Labor reported that the growth rate of Internet correspondence training participants was 
6,281% over the past six years (19,653 in 1999 and 1,254,066 in 2005) (Ministry of Commerce et al., 2006). While 
this astronomical figure can be explained in many ways (Lee, 2006) with positive results, its negative effects on 
corporate e-leaming and distance learning have also been noted. 

This study intends to discuss the causes and effects of the rapid growth, as well as explore directions for 
future research and practice with regard to corporate e-learning in Korea. To determine the current issues, the study 
focuses on the significant developments implemented by a government- funded research center, the Korean Research 
Institute for Vocational Education & Training (KRIVET), over the past six years (2000-2005). KRIVET impact on 
the development of corporate e-leaming can hardly be overstated (Lee, 2006). The causes of those developments and 
various resulting problems are examined in this paper. Finally, based upon this analysis, future directions are 
suggested. 



II. Current developments in corporate e-learning in Korea: 2000 - 2005 

Corporate e-learning in Korea developed in earnest following the implementation of a new government 
policy in 1999 on Internet correspondence training. Since then, the development of the field can be measured in five 
ways: rapid quantitative growth, government initiatives, dominance of the tutorial mode, quality assurance, and 
high adoption rate among large corporations. These will be discussed in turn. 

Rapid quantitative growth 

One of the salient features of corporate e-learning in Korea over the past years has been its rapid 
quantitative growth. In Table 1 , it may be observed that the number of employees who participated in the Internet 
correspondence training program or e-learning greatly increased from 19,653 in 1999 to 1,254,066 in 2005 
(approximately 63.8 times more). In 1999, only 2.5% of the total number of trainees participated in the Internet 
correspondence training. But in 2005, that ratio increased up to nearly half of the total number of trainees (46.4%) in 
2005. 
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<Table 1> Number of Trainees Supported by Employment Insurance Act( 1999 -2005) 



Year 


Internet-Based 


Postal-Based 


Classroom-Based 


Total 




Internet 

Trimmer Rqtp 


Trainees 

(A) 


Increase 

rate 


Trainees 


Increase 

rate 


Trainees 


Increase 

rate 


Trainees 

(B) 


Increase 

rate 


-L 1 Ivdlv 

(A/B*100) 


1999 


19,653 


- 


85,055 


- 


676,700 


- 


781,408 


- 


2.5% 


2000 


137,712 


600.7% 


161,825 


90.3% 


920,797 


36.1% 


1,220,334 


56.2 


11.3% 


2001 


406,159 


194.9% 


199,242 


23.1% 


950,001 


3.2% 


1,555,402 


27.5 


26.1% 


2002 


543,320 


33.8% 


197,045 


-1.1% 


843,958 


-ll.%2 


1,584,823 


1.9 


34.3% 


2003 


629,930 


15.9% 


193,570 


-1.8% 


838,478 


0.7% 


1,661,978 


4.9 


37.9% 


2004 


929,771 


47.6% 


283,338 


46.4% 


790,354 


-5.7% 


2,003,463 


20.6 


46.4% 


2005 * 


1,254,066 


34.9% 


339,645 


19.9% 


1,171,630 


48.2% 


2,705,341 


35.0 


46.4% 



Source: Ministry of Commerce et al. (2006). E-learning White Paper 2005-2006. 

* The number of trainees in 2005 is projected on the data from the Ministry of Labor in Oct. 2005. 

The high growth was due to the expansion of the Employment Insurance Act , which earmarked financial 
support for e-learning programs. The law allowed the Ministry of Labor to begin in 1999 providing institutional 
support. Thus, the number of corporations and workers that participated in e-leaming increased rapidly over the 
previous 5-6 years. Moreover, corporations took advantage of e-learning, giving more employees access to 
educational opportunities at relatively low cost. The large corporations quickly adopted the e-leaming systems and 
invested money to develop programs. 

The rapid growth of corporate e-leaming in Korea can be also attributed to an increase in theoretical studies 
on corporate e-leaming. One of the leading journals on corporate training in Korea is the Journal of Corporate 
Education. This academic journal published its first volume in 1998 in the area of training methods, focusing 
especially on the applications of different technologies and programs. Not surprisingly, the journal has discussed e- 
leaming with regular frequency since 1999. Of the 13 volumes of the journal, 26 of 79 articles (33%) have dealt 
directly with the subject of corporate e-leaming. Considering the plethora of educational methods and issues in 
corporate training that could be discussed, devoting over 30% of the journal articles to e-leaming is doubtlessly a 
testament to the rapid quantitative growth of the field as well as the high level of theoretical interest it has garnered 
in Korea. 

Government initiatives on corporate e-learning 

The Korean government, especially the Ministry of Labor, has played a significant role in developing the 
field of corporate e-leaming. In accordance with the government’s strategic plan hatched in the mid-1990s to 
transform the country into a knowledge-based information society, the Ministry of Labor has been the primary 
driving force behind the implementation of e-learning for corporate training since 1998. That year the Ministry of 
Labor initiated a pilot project that tested e-leaming based training courses, a project which led to the conclusion that 
correspondence training should include both Internet correspondence training and postal correspondence training 
and resulted in expansions the following year. This decision caused both corporate e-leaming and distance learning 
for adults to grow rapidly in Korea in 2000 (Table 1). Furthermore, as alluded to previously, the Ministry of Labor 
established a special division, the e-Learning Center at KRIVET, to monitor the quality standards of the e-leaming 
institutes and of the Internet correspondence training program as well as to establish a new support system and make 
recommendations . 

In addition to these early initiatives of the Ministry of Labor, subsequent measures and policies have also 
been effected since 2000 to boost corporate e-learning (Lee, 2006). In order to support corporate e-leaming on a 
long-term basis, the Ministry of Labor and the e-Leaming Center at KRIVET developed the Corporate E-Learning 
Mid-Period Development Plan, 2004 - 2008 , which proposed strategies covering various aspects of e-leaming such 
as servicing and maintenance of the system, infrastmcture constmction, cultivation of human resources, 
standardization, and quality control. Recently, Internet correspondence training regulations have also been revised to 
incorporate ‘blended learning’ and a scaled training fee stmcture, based on an analysis of the quality of the e- 
learning institutes. Lastly, the worker’s tuition support system has been modified as well so that individual workers 
can receive financial support when they register independently for e-learning programs. 
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Dominance of tutorial e-learning 

Most corporate e-leaming courses in Korea are in tutorial format in which major points are supplemented 
with elaborations and examples (‘Intro’ and ‘Lesson’ in Figure 1) followed by practice problems (‘Activity’ in 
Figure 1). Various design strategies have been applied — for example, Keller’s ARCS (Attention, Relevance, 
Confidence, Satisfaction) model has been adopted to enhance effectiveness and for animated presentations advanced 
technologies such as Flash have become standard authoring tools. 




[Figure 1] An example of e-Learning: Tutorial type 



Table 2 shows that nearly 90% of all e-learning content in 2005 were tutorials in either HTML 1 or Lecture- 
on-Demand (LOD) format. The remaining 10% were simulations that honed technical skills. As these two types — 
tutorials and simulations — became so ubiquitous over the past three years in the field of corporate e-learning, the 
government’s Employment Insurance Fund eventually dropped support for other types of e-leaming programs. 



<Table 2> e-Learning Program Proportion by Types from 2003 to 2005 





2003 


2004 


2005 


Types 


Self developing 


Outsourcing 


Self 

developing 


Outsourcing 


Self 

developing 


Outsourcing 


HTML 

Tutorial 


545 (88.0) 


1,086 (79.3) 


738 (88.0) 


1,005 (68.6) 


742 (79.7) 


1,082 (80.8) 


LOD 


26 ( 4.2) 


212(15.5) 


33 (3.9) 


315(21.5) 


83 ( 8.9) 


138(10.3) 


Simulation 


48 ( 7.8) 


72 ( 5.3) 


68(8.1) 


146(10.0) 


106(11.4) 


119(8.9) 


Total 


619 


1,370 


839 


1,466 


931 


1,339 


(100.0%) 


(100.0%) 


(100.0%) 


(100.0%) 


(100.0%) 


(100.0%) 



Source: Lee, S. et al. (2006). Evaluation of Internet Correspondence Training Institutes, Year 2005. Seoul: KRIVET 



Quality assurance 

The government, the Ministry of Labor in particular, has tried to assure e-learning quality over the previous few 
years. In 2002, the e-Leaming Center at KRIVET launched an assessment system to judge the appropriateness of Internet 
correspondence training programs. All e-leaming programs supported by the Employment Insurance Fund were required 
to be evaluated by the Center. As seen in Figure 2, the B level grew annually from 16.5% in 2002 to 48.1% in 2005, 
demonstrating that instmctional design and content quality were indeed improved. Conversely, the proportion of D and F 
level steadily decreased from 27.5% in 2002 to 5% in 2005. The quality of the e-learning programs was successfully 
improved by implementing the assessment system. 



1 HTML in this Table refers to Web-based instmction, in which text, graphics, and some animated objects are 
displayed in HTML format. LOD(Lecture-on-Demand) refers a type of e-Learning in which lectures by an instmctor 
are recorded and delivered through the Internet at learner’s convenience. 
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(unit: %) 
80 — 



— A 
— B 

— D 
— Fail 



2002 2003 2004 2005 

[Figure 2] e-Learning Program Assessment Results( 2002 - 2005) 

Source: Ministry of Commerce et al. (2006). E-Learning White Paper 2005- 2006. 

High adoption rate in large companies 

Participation in corporate e-learning programs has not been evenly distributed across the corporate sector. 
The employees of larger corporations comprise the largest proportion of Internet correspondence training system 
participants. Table 3 shows that in 2004 just 8% of training participants from the assembly line in small and medium 
companies participated in Internet correspondence training programs, while nearly 30% of those from large 
companies did so. 




Company 



<Table 3> The Proportion of Training Types for Assembly Line Workers by Company Size 

(Unit: number, person) 

Small & Medium 

Company Field training Total 

Consortium 



Classroom 

Training 



Internet 

Training 



Postal 

Training 





Programs 


Trainees 


Programs 


Trainees Programs 


Trainees 


Programs 


Trainees 


Programs 


Trainees 


Programs 


Trainees 


Small and 


1,847 


14,432 


432 


1,877 


300 


6,386 


3 


22 


8 


90 


2,590 


22,807 


Medium 


(71.3%) 


(63.27%) 


(16.8%) 


(8.2%) 


(11.6%) 


(28%) 


(0.1%) 


(0.1%) 


(0.3%) 


(0.4%) 






Large 


4,561 


68,565 


1,145 


34,294 


865 


14,401 


1 


1 


4 


6 


6,576 


117,267 


(69.4%) 


(58.5%) 


(17.4%) 


(29.2%) 


(13.2%) 


(12.3%) 


(0.02%) 


(0%) 


(0.1%) 


(0%) 







Source : Ministry of Labor, Unemployment insurance DB, 2004. 



Table 4 further shows that the implementation ratio decreases significantly in relation to smaller company 
size. In small companies (less than 50 employees), only 10.3% of workers participated, whereas 52.4% of 
employees did so in large companies (140-299 employees) (Jang & Yoo, 2006). 



<Table 4> e-Leaming Program Proportion Change by Company size in 2006 



Category 




Company size 




total 


difference 


Less than 

Workers 


50 50 Workers to Less 

than 149 Workers 


150 Workers to Less 
than 299 Workers 


e-Leaming Yes 


8 ( 10.3) 


15(24.6) 


22 ( 52.4) 


45 ( 24.9) 




Program No 


70 ( 89.7) 


46 ( 75.4) 


20 ( 47.6) 


136(75.10 


X =23.938 


total 


78 (100.0) 


61 (100.0) 


42 (100.0) 


181 (100.0) 


p<.01 



Source: Jang, M. H., & Yoo, S. J. (2006). E-Leaming Needs Analysis for the Employees of Medium and Small Companies. 



Journal of Vocational Education & Training , 9(3), 175-202. 

The Internet correspondence training system appears at first glance to provide broader access to educational 
opportunities in Korea. It seems to put training within reach of those employees who would not otherwise have such 
an opportunity. However, in realty, it fails to achieve that goal. Assembly line workers and those employed by small- 
or medium-sized companies have not received equal opportunities for the Internet correspondence training. 
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III. Issues in corporate e-learning in Korea 



Although there have been great developments in corporate e-leaming in Korea over the past six years, there 
remain certain key problems in the field: moderate quality, uniformity and controlled growth, limited evaluation 
criteria, and unevenly distributed adoption rates. 

Moderate quality of corporate e-learning 

The quality of corporate e-leaming has become an issue that has only been compounded by the rapid 
growth rate. As mentioned, since the 1999 introduction of e-learning in Korea two formats have emerged as 
preeminent: one is LOD (Lecture- on- demand) in which the presentation of a lecturer is recorded in motion picture, 
and the other is Web-based instmction in which text, graphic, and some animated objects are displayed on Web 
pages as the learners click to progress. LOD has become a dominant format because not only it was easy and 
relatively cheap to develop but it also conformed to expectations of what training should be (i.e. lecture delivery at 
learners ’ convenience ). A small percentage of e-leaming was developed as web-based instmction and most of this 
type were just copies of printed materials without learners’ active participation. These problems were partially 
remedied when the new assessment system was introduced in 2001. As Table 5 illustrates, growth stalled in 2002 
(the number of total institutes decreased from 1 10 to 93) with the introduction of governmental regulations, of which 
the assessment system was a key provision. The system succeeded in upgrading the quality of tutorial-type e- 
leaming programs at the expense of other formats. The quality issue will be discussed in more detail in the latter 
portion of the quality assurance section. 



<Table 5> Number and Increasing rate of Corporate e-Learning Institutes in Korea( 2001 - 2005) 





Self-developing 


Outsourcing 




Total 


Year 


Institutes 


Increasing 

rate 


Institutes 


Increasing 

rate 


Institutes 


Increasing 

rate 


2001 


56 


- 


54 


- 


110 


- 


2002 


57 


1.8% 


36 


-50.0% 


93 


-18.3% 


2003 


55 


-3.6% 


33 


-9.1% 


88 


-5.7% 


2004 


53 


-3.8% 


55 


40.0% 


108 


18.5% 


2005 


68 


22.1% 


80 


31.3% 


148 


37.0% 



Source: Ministry of Commerce et al. (2006). E-learning White Paper 2005-2006. 

Notes: 1 . If a company provides both intra-company and outsourcing programs, it will be estimated as outsourcing training. 



2. The number of companies conducting more than one training per year. 

Uniformity and controlled growth 

The government initiative on corporate e-learning in Korea has yielded unexpected results. Although it 
stimulated the quantitative increase of corporate e-learning programs in a short period of time, one mode of e- 
leaming, tutorial, dominated the e-leaming landscape. While the Evaluation System for Contents and Design Quality 
of e-Learning by the e-Leaming Center ( Table 7) succeeded in upgrading the basic quality of the tutorials, it failed 
to encourage the development of diverse modes of e-leaming programs beyond simple tutorials, and also neglected 
to support new ideas and studies to help e-learners become self-regulated or self-directed(Lim, 2005). Many e- 
leamers did not successfully complete the e-leaming programs in which they had enrolled; they dropped off early on 
or midway through because they were not self-motivated. 

Although tutorial is an effective instmctional type for certain objectives, it cannot support some essential 
objectives and activities, including teaching problem-solving skills, creative thinking skills, and self-directed 
learning. Yet these are the very skills and experience expected of workers in Korea’s knowledge-based society. 
However, corporate e-leaming in Korea currently has not successfully promoted different e-learning models such as 
Problem-Based Learning, Goal-Based Scenario, and Case-Based Learning (Kang et al., 2006). 

In addition, the government, by virtue of their initiative, regulated the growth of e-leaming in a way that 
stifled spontaneous innovation. Most companies assumed a passive role in designing and developing other types of 
e-leaming. The rapid growth of e-leaming was achieved without autonomous efforts from the corporate sector. For 
instance, the possibility of integrating e-learning with the knowledge management system in a company (Rosenberg, 
2001) or with long-term blended learning strategies has not been systematically examined, in spite of the high 
demand. 
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On the other hand, some alternative designs have been explored from a theoretical standpoint. The Journal 
of Corporate Education in Korea has treated many topics related to course design , and secondly, to learner support 
and cost issues when those articles from the journal were analyzed by an analytical framework of Rha and Han 
(2002) 2 (Table 6). As this journal mainly focuses on the educational methods in corporate training, especially from 
the perspective of educational technology , this is not an unexpected result. The course design topics covered in the 
articles included ‘strategies for learning motivation in e-leaming,’ ‘design strategies for Goal-Based Scenarios,’ 
‘blended learning strategies’, and ‘A design model for e-leaming.’ On the other hand, as e-learning is relatively new 
to adult workers, learner support issues have been widely discussed. The issue of cost has also been examined 
because the corporate sector has been particularly interested in determining whether e-leaming is cost-effective. 



<Table 6> The Topics of e-Learning Studies in Korea (1998 -2006) 



Topics 


Frequency 


Ratio 


Course Design 


11 


42.3% 


Learner support 


6 


23.1% 


Cost 


4 


15.4% 


Learner participation 


3 


11.5% 


Learning Contents 


1 


3.8% 


Other 


1 


3.8% 


Total 


26 


100.0% 



Note: The articles from the Journal of Corporate Education in Korea were selected for examination 

In addition to the Journal of Corporate Education other academic journals and periodicals have also 
discussed topics related to e-learning design. The Journal of Educational Technology in Korea, for example, has 
dealt with issues of corporate e-learning from the perspective of educational technology, notably, issues such as 
tutoring (Cho & Lee, 2004) and supporting self-regulation (Lim, 2005) have been areas of recent focus. 

Limited evaluation criteria 

It can be argued that the current perception of quality of corporate e-leaming can be attributed to the narrow 
and ambiguous evaluation criteria. Table 7 shows the current evaluation criteria for e-learning programs. It consists 
of five dimensions: instructional design, interaction, evaluation, instructional support design, and technology. 
Whereas the dimensions and the sub-criteria do cover important quality aspects of e-leaming programs, some 
problems still exist. First, they focus too narrowly on evaluating the tutorial type of e-learning, virtually assuming 
that the tutorial format is the only one to assess; other types of e-leaming programs such as Case-Based Learning or 
simulation have not been easily evaluated under these criteria. Second, they do not provide specific guidelines for 
each criterion. The criteria were ambiguous and judgment could vary depending on the evaluator. More specific 
guidelines in a mbric format should be developed to make the evaluation more objective and effective for future e- 
leaming programs. 



<Table 7> The Evaluation Criteria for e-Leaming Programs 

Dimensions Criteria 

Instruction Design Content presentation strategy and method relevancy 

Suitability for supplementary & advanced learning resources 
Interaction Appropriateness of learner- instructor interaction 

Appropriateness of learner-learner interaction 
Appropriateness of learner- contents interaction 



2 

The framework suggests course design, learner support, cost, learner participation, and learning contents as the 
major components of e-Leaming. Course design includes ‘the systematic use of instmctional strategies’, ‘the 
flexible design and change of course’, ‘active interaction’. Learner support deals with ‘easy admission and 
graduation’, ‘counseling system’, and ‘stable infra stmcture and technology’. Cost involves such areas as Tow 
tuition fee’ and ‘cost-effectiveness’. Learner participation examines ‘learner involvement’, ‘learner satisfaction’, 
and ‘free choice of course’. Finally, learning contents deals with ‘interesting content’, ‘easiness to accessing 
information’, and ‘usefulness of content’. 
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Appropriateness of learner-program manager interaction 
Evaluation Appropriateness of method 

Instructional Support Design Appropriateness of help function 

Connectivity of navigation (Interface) 

Technology Technical stability 

Source: e-Leaming Center in KRIVET (2006) 

Unevenly distributed adoption rates 

As noted previously, the implementation of corporate e-leaming in Korea has mostly been confined to 
larger companies. Employees of small- and medium-sized companies have not yet fully experienced e-leaming 
programs. To overcome this shortfall, the Ministry of Labor’s worker’s tuition support system has been recently 
revised to provide financial support for these underserved workers (Lee, 2006). Further consideration in terms of 
more access to Internet correspondence training system is needed in order to improve corporate e-leaming in Korea. 

From a theoretical vantage point, selected studies in the Journal of Vocational Education & Training by 
KRIVET have discussed the issue of supporting medium and small companies, such as by beginning with an e- 
leaming needs analysis of these companies (Jang & Yoo, 2006) and an assessment of e-leaming course selection 
criteria (Kwon, Lee, Rha, & Lim., 2006). These studies were mainly conducted to come up with political 
implications for the future of e-leaming implementations at the national level. 

IV. Conclusions: Future directions and prospects 

The current development of corporate e-learning in Korea was examined in terms of five aspects: Rapid 
quantitative growth, government initiative, dominance of the tutorial mode, quality assurance, and high adoption 
rate among large corporations. Each one of these also has a corresponding weakness: Rapid quantitative growth has 
meant that quality has at best been moderate. Government initiative has fueled complacency as companies have 
hesitated to develop their own e-leaming programs for specific purposes such as problem-solving or creative 
thinking skills. The dominance of the tutorial mode has kept other types of e-leaming from being actively designed 
and implemented. Quality assurance measures have used evaluation criteria so narrow and ambiguous that other 
types of e-leaming contents could not be easily evaluated and the evaluation was susceptible to subjective influence. 
And finally, the high adoption rate among large corporations has translated into workers of small and medium 
companies not having equal opportunities. 

Nevertheless, corporate e-learning does seem to have had a great impact on the recent development of life- 
long education and distance education in Korea. While the ratio of participants in life-long education in Korea has 
been relatively low among OECD countries (OECD, 2000), corporate e-leaming has played an important role in 
increasing the rate of participation rate in a short period of time. This increase was mainly due to the government 
initiative to transform the state into an information-based society where all the aspects of government, including 
education and training systems, rely on information infrastmcture and environments. The Ministry of Labor was no 
exception, and it took advantage of the Employment Insurance Fund to achieve its goal for contributing to the 
information-based society. Companies were permitted to ask for government subsidy as long as they provided e- 
leaming programs for their employees. Implemented in 2000, this allowance had a significant effect on the growth 
of corporate e-learning. 

In this respect, corporate e-learning in Korea has a unique distinctive. The main impetus for its rapid 
growth has not been that companies needed to provide high-quality training programs to more workers but that the 
government took initiative to transform the nation into an information-based society. Most e-leaming programs were 
tutorials that could be made easily, and their common objectives were for acquiring knowledge or understanding 
content(Lee et. al., 2006). Companies wanted to ensure that employees could do their work competitively or show 
competency in new areas. They did not want any more workers who just understand basic, factual knowledge. 
However, competency-related e-learning or performance-based e-learning has not been fully examined yet. 

Corporate e-leaming can be improved in two ways. First, it should be dynamic rather than static, serving 
the new requirements of companies and employees. If it remains confined to traditional modes of education or 
schooling where understanding knowledge can be acceptable as an educational goal, it will be criticized and 
eventually phased out. It should be responsive to demands for new skills, competencies, or performance, training 
objectives that can improve employee effectiveness and efficiency. Therefore, the current government e-leaming 
initiative should be changed into one that is directed by both government and companies , one in which companies 
play an active, executive role and have a vested interest. Each company should be encouraged to develop and 
implement advanced and authentic programs autonomously, programs such as an e-leaming version of Problem- 
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Based Learning or Case-Based Learning (Kang & Oh, 2006). And the evaluation criteria should become more 
inclusive and go beyond examining the components of the tutorial. It should stimulate new trials and developments 
of corporate e-learning programs. 

Second, corporate e-leaming is a kind of distance learning. Theoretically this provides more possibility to 
help adult learners access education and training opportunities. Corporate e-leaming can provide training to 
employees who might otherwise never have had the opportunity to train because of their time and space constraints. 
The initial implementation of corporate e-leaming was geared toward large corporations such as Samsung, LG, and 
SK. These were relatively eager to adopt the program because they considered e-leaming to be cost-effective and to 
provide more training for their employees(Jang & Yoo, 2006). Yet small- and mid-sized companies were not able to 
offer these programs due to financial constraints. Moreover, they did not recognize the value of e-leaming for their 
workers. To solve this inequality problem, the government should play a key role in encouraging the implementation 
of e-learning. The Bureau of Small- and Mid-Sized Companies in the Ministry of Labor must come up with effective 
strategies to mitigate the imbalance, while the consortium of affected companies should take ownership of 
implementing e-learning. And, government and companies should collaborate to provide leadership aimed at 
providing opportunities for the disadvantaged employees of these companies. 

As mentioned, the unequal access to e-leaming should be acknowledged and countermeasures to provide 
more equal opportunities should be devised, especially for employees for whom e-Leaming is relatively new 
(Piskurich, 2003; Lim, 2005). Considering their schooling experience that focused on passive learning, it is easier to 
see how it may be difficult for such employees to manage their independent learning, or exercise appropriate self- 
direction or self-regulation(Lim, 2005) in learning. Providing facilitator intervention or even a new learning 
management system to encourage and support learner’s self-regulation systematically are steps that could mitigate 
the problems. This issue in terms of supporting learner’s self-direction or self-regulation should be studied and 
examined in further research. 
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Abstract 

Wikis, when used as an open editing tool, can have profound and subtle effects on students’ 
collaborative learning processes. Do students feel comfortable editing each others’ wiki articles? How are 
collaborative learning outcomes impacted when students communicate using wikis? This study addressed 
these issues using qualitative methods, including multiple semi-structured interviews and student reflective 
journals for analysis. The findings challenge idealistic hypotheses that wiki work, without careful design 
and implementation, is naturally beneficial. It was also found that collaborative writing and learning were 
the exception rather than the norm among participants. It is recommended that instructors provide highly 
supportive learning experiences to teach students how to use wikis and how to work collaboratively when 
implementing wikis to maximize the benefits of this emerging tool. 

Introduction 

Wikis have been lauded in the literature as being an ideal tool for collaborative learning because 
learners are able to write, edit, version, and discuss next to the content. According to Holmes, et al. (2001) 
theory of communal constructivism, wikis can be used to move learners toward a state of communal 
constructivism , where students and teachers are not simply engaged in developing their own information 
but actively involved in creating knowledge that will benefit others, hence, constructing knowledge for the 
community, not just the self. 

Given the promise of wikis for open editing and collaborative learning, the question remains: do 
students take full advantage of the collaborative features inherent in wikis, which are not available in e-mail 
and discussion boards? Many authors have discussed the functionality of wikis (Desilets, Paquet, & 

Vinson, 2005; Fuchs-Kittowski & Kohler, 2005; Schaffert, et al. 2006); however, little explicit attention 
has been paid to the human interactions surrounding students’ use of the collaborative features of wikis to 
maximize learning outcomes. How do students interact, communicate, and collaborate to achieve learning 
goals in wikis? 

This study addressed the need for empirical evidence regarding student interactions when working 
collaboratively in a wiki by qualitatively exploring the evolving phases of using wikis to complete class 
assignments. The purpose of this study was to investigate 1) how students communicated in wikis, 2) 
students experiences in writing and editing collaboratively in an open writing environment, and 3) if 
students learned collaboratively. 



Review of the Literature 
Computer-Mediated Collaborative Learning 

Brandon and Hollingshead (1999, p. 54) defined collaborative learning as “an activity that is 
undertaken by equal partners who work jointly on the same problem rather than on different components of 
the problem.” Moallem (2003, p. 84) stated that “While learning is ultimately an individual enterprise, the 
support of a group with a common learning objective can produce a synergistic facilitation of learning by 



188 




each member of that group.” Similarly, Holmes, et al. (2001, p. 1) considered that collaborative learning 
was “an approach to learning in which students not only construct their own knowledge as a result of 
interaction with their environment but are also actively engaged in the process of constructing knowledge 
for their learning community.” A meta-analysis (Johnson, Johnson, & Smith, 1991) of the comparative 
effects of collaborative learning and individual learning concluded that college students improved their 
higher order thinking skills such as reasoning, idea generation and solutions, more in a collaborative 
environment than working solo. 

With the increased use of the Internet in educational settings, computer-mediated collaborative 
learning has attracted much attention. Computer-mediated collaborative learning takes advantages of 
Internet resources and online communication tools (e.g., discussion board, e-mail, conferencing systems, 
videoconferencing, and chat rooms) to facilitate communication and collaboration. In such environments, 
students can access sufficient and just-in-time resources, reorganize their thinking, present new forms of 
knowledge, and be exposed to multiple views from groups (Uribe, Klein, & Sullivan, 2003). 

Empirical studies that have investigated the effectiveness of computer-mediated collaborative learning 
in classrooms have proliferated in the literature. A meta-analysis (Lou, Abramin, & d’Apollonia, 2001) 
study examined 122 studies and found that small groups learning with technology outperformed individuals 
learning with technology. In addition, researchers examined whether certain instructional methods worked 
effectively in a computer-mediated environment. Johnson and Chung (1999), Mergendoller et al. (2000), 
and Uribe, Klein, and Sullivan (2003) found that computer-mediated environments were not only suitable 
for project-based learning but also contributed to higher order learning. Similarly, Jonassen, Prevish, 
Christy, and Stavrulaki (1999) and Murphy and Collins (1997) investigated whether using case studies in a 
collaborative environment helped learning transfer from a face-to-face environment to computer-mediated 
environment. Their results support the hypothesis that case studies work well in a computer-mediated 
environment and enhance higher order learning. 

The social dimensions of learning at a distance have also been investigated and were found to be vital 
to learning, and yet often the missing element in distance education courses (Berge, 1999; Moore & 
Kearsley, 1995; Spitzer, 2001). These authors argued that if online interaction is not an integrated and 
expected component of online learning, students probably will not use the available communication tools. 

In general, research has provided evidence that a computer-mediated environment supports 
collaborative learning and enhances learning outcomes. Along these lines, educators place greater 
expectations on the potential role of wikis as a recent computer-mediated communication tool to enhance 
collaborative learning in education than the research can support. 

Pedagogical Claims of Wikis 

Wikis have been credited with increasing democracy within groups by focusing on the community 
rather than the individual, thus, increasing students’ skills in collaborative work (Holmes, et al., 2001). 
Collaboration is considered beneficial because learners must consider multiple perspectives, can rely on 
each other to reduce uncertainty during complicated activities, and increases participation in the learning 
activity. By using wikis, students’ learning is extended over time, across collective environments and 
people, and through open-natured projects that create public goods (Ciffolilli, 2003). Wikis have the 
potential to maximize writing, reflecting, reviewing, and witnessing cumulative results (Fountain, 2005; 
Hammond, 2005). Fuchs-Kittowski and Kohler (2005, p. 2) reported that wikis support “cooperative 
community knowledge generation” by allowing users to create and maintain a “community knowledge 
base, offering a quick, simple way to produce and review information that can be gathered and linked to 
other wiki pages.” 

While researchers and educators laud the use of wikis for supporting collaborative and inquiry-based 
learning (Engstrom & Jewett, 2005), improving critical thinking skills, and “empowerment of the 
individual in terms of life-long learning” (Freeman, Holmes & Tangney, 2001, p. 1271), little empirical 
research has been conducted to test the pedagogical promise of wikis (Edwards, 2002; Freeman, Holmes & 
Tangney, 2001; Hammond, 2002). The majority of the literature surrounding the use of wikis in 
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educational settings is based on “speculative and aspirational stances rather than strong theoretical or 
empirical grounds” (Edwards, 2002, p. 1). 

Applications of Wiki in Classrooms 

Engstrom and Jewett (2005) studied 400 students with 1 1 teachers who used wikis for an inquiry-based 
project in a professional development teacher-preparation program. They found that students posted and 
edited insufficiently in their small research groups, which primarily reflected surface-level learning in the 
inquiry process and that the teachers did not model or facilitate an exchange of ideas, questions, or provide 
feedback to students required to build wiki pages. The study suggested that project developers model the 
wiki practice that prompts interaction, critical thinking, and thinking from multiple perspectives. Wang et 
al. (2005) investigated students’ editing behavior and their performance in wiki. They explored the effects 
of students’ frequency of editing usage in wikis on their final exam performance and found that students 
with low usage on the wiki performed better than those with high wiki usage on the final exam. 

Other studies investigated factors such as online environment and anonymous writing that affected 
effective collaboration. Bold (2006) incorporated wiki in a Master’s online courses to support 
collaboration. Although the study did not survey students on wiki use specifically, it investigated whether 
the use of wiki facilitated online interaction. The study indicated that wiki put students in charge of jointly 
posting, editing, reporting, maintaining work without burdening individual students as project coordination. 
In this manner, students reported that the use of wiki increased their sense of connection in the online 
setting. Chong & Yamamote (2006) investigated a group of 20 people who had not met prior to the study, 
yet felt comfortable in exchanging ideas in a wikis writing project. Their study suggested that anonymous 
writing gave students’ a private space, therefore, collaboration between strangers could facilitate 
independent thinking, clear understanding of the team members, thus contributed to high quality outputs. 

Hewitt and Peters (2006) asked their 1 5 students to construct wiki pages on an array of topics in a 
graduate-level distance education course. They found that the students considered building a knowledge 
base in a wiki project was not only an authentic task for themselves but also a value-added activity to future 
classes. Coutinho and Bottentuit (2007) asked 16 students in a Master’s program to build an enormous 
collaborative repository that could help future students with their dissertation. Their findings were 
inclusive. On the one hand, students enjoyed working with they group. On the other hand, they did not 
think team work has a better quality or helped them learn more than if they worked by themselves. 
Similarly, Plowman (2007) provided his students with a wiki environment where they built a social justice 
forum. Through the very interactive forum, the participants went through the circle of construction - 
deconstruction - reconstruction several times so that their knowledge building experience was not linear. In 
contrast, Blank, et al. (2004) reported students’ territorial needs prevented true collaborative work in a wiki 
environment. 

These studies lay the groundwork for this study, which addressed the gap in the literature regarding 
how students communicated, whether communication affected their wiki writing, and whether the potential 
of wikis - effective collaborative learning - had been achieved. 



Methodology 

Context of the Study 

The study was situated in a graduate level course about adult education, offered at a major land-grant 
university in the mid- west. The instructor built the shell for a wikibook using Mediawiki® software 
( www.mediawiki.org ) and posted it online, purposefully public and accessible. 

Students ( n=5 ) were required to write four articles using wiki as the writing and presentation tool. 
Students were to work individually on two chapters, collaboratively on two chapters, and were asked to edit 
another student’s article for the fifth assignment. In addition, students were given a detailed rubric for 
completing the assignments along with face-to-face instruction about the project. Students could create a 
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chapter under the instructor-defined headings or create a new article. Moreover, the teams were assigned by 
the instructor to increase heterogeneity in regard to culture, age, and discipline. The instructor met with 
student teams weekly to access progress and cohesiveness of the team. 

Instrumentation 

An original semi-structured interview protocol was created and reviewed by three researchers. 
Interview questions were developed based on Holmes et al. (2001) theory of communal constructivism. 

Data Collection 

Data were collected from four sources: the contents of the wiki, student interviews, students’ weekly 
reflective journals, and meeting notes taken by the researchers in the classroom. Specifically, a total of 
eight interviews were conducted face-to-face, three of which were team interviews after two collaborative 
wiki articles were completed. The rest of the interviews were conducted one-on-one after students 
submitted their individually created wiki articles and finished the edits of one wiki article created by others. 
In addition, meeting notes taken by the instructor and the researchers as well as students’ reflective 
journaling of their learning experiences were also part of the data set and were analyzed qualitatively. 

Validity, Reliability and Objectivity 

To ensure the reliability and accuracy of the findings, the researchers followed Lincoln and Guba’s 
(1985) framework of trustworthiness. Internal validity was achieved by insuring congruence between the 
findings and reality. A synthesis of each interview was prepared and sent to all the participants for content 
verification (member checking the data). In addition, multiple interviews were conducted and served to 
verify previous statements (triangulation). External validity was achieved by transferability of research 
findings to other situations. Thick (detailed) description of structures and processes were revealed by the 
data and articulated for adaptation (audit trials). 

Reliability, or replication of research findings is not a goal of qualitative studies, but detailed field 
notes and memos were maintained throughout the process and serve as an audit trial to verify findings. In 
addition, students’ reflective journals were collected and analyzed for triangulation purposes. Objectivity 
was achieved by creating a research team (course instructor and external interviewer and researcher) for 
peer examinations purposes. Data management and transcribing process were carefully maintained. 
Interviews were transcribed verbatim, and write-ups of process notes and interview summaries were 
documented and verified by the research team. 



Results 



Phase 1: Exploration (A Crisis of Authority) 

Communication. The participants indicated that using an open editing tool such as wiki was difficult at 
the beginning of the project because it challenged their notions of group work. The primary difficulty 
involved ineffective communication within the group such as not knowing what to communicate to each 
other, frequency of communication, or how to use the wiki discussion board to communicate with. 
Participants 1 noted, (Excerpt 1) “the first chapter was chaotic. We had the technology problems because 
things wouldn’t get posted in time, and because we didn’t know how to post it. We had to view the 
discussion board and there was a lack of communication because we weren’t all on the same page.” 
Participant 2 said (Excerpt 2) “at first it was hard to get it right. I didn’t know how to write [in the wiki].” 

A lack of communication resulted in students doing separate work for a supposedly collaborative 
project. Participant 1 said (Excerpt 3) “for the first group project, our group didn't communicate as much as 
we should have. When we got together, we just showed everybody our work.” Participant 6 said (Excerpt 
4) “I can just do the chapter on Word, but to put it in the wiki format. It took a while for me to know [how 
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to use the wiki].” Thus, students created work individually, then put their writing together at the end of the 
project rather than the idealized co-writing that leads to improved critical thinking. 

Writing. When asked about the impact of writing in the wiki in terms of increasing learning outcomes 
participant 1 said (Excerpt 5), “when developing this chapter, I’m not sure I learned a lot. It was mostly 
compiling all the information we had learned over the semester into a practical and useable format.” When 
asked if they felt comfortable about changing and editing other students’ work, all the participants were 
unanimous. They all said “not really” after the first project was completed. Participant 3 said (Excerpt 6) “I 
don’t know if I would like to change another person’s work. I could do some editing, but I would want to 
go over things with them first. I figure that I should go to the individual person quietly first [before 
editing].” When asked further what made them uncomfortable, participant 1 stated, (Excerpt 7) “because 
it’s their work, because it’s their tasks, and you’ve been taught to work on your own work. And it’s hard to 
kick that habit.” 

The participants’ comments indicate that they were hesitant in editing others’ articles when the project 
began. The discomfort resulted from a feeling of a lack of ownership in the article and they needed time to 
explore feelings prior to editing, indicating a crisis of authority in co-creating a joint article. Students saw 
the assignment as pieces of a pie to be delegated and reassembled after the fact, rather than a whole pie in 
which they would all take credit for. 

Collaboration Compared with first generation web tools such as e-mail and discussion boards, wikis 
are considered superior second generation web-based social tools because of the increased capacity for 
collaborative work. Collaboration; however, did not occur in the first phase of the project where students 
were asked to co-write an article in wiki. Rather, students divided the assignment into smaller parts, 
completed their part, and cut and paste the article together after it was written individually. The wiki was 
used as a presentation tool rather than as a writing tool and the results were awkward at best. Participant 1 
said (Excerpt 8), “at first, I didn’t really want to use wiki. It really intimidated me, and so I did everything 
outside of it and just cut and paste [my finished article into wiki]. Participant 6 notes that (Excerpt 9) “you 
can feel the rough transition between each part [because it was created as separate pieces and merged at the 
end]. There's not a good flow in the chapter.” 

None of the students had used a wiki prior to this assignment. During the early phase of the assignment 
the participants had difficulty communicating with each other in a timely manner. The participants also felt 
high anxiety and uncertainty about authorship issues and feared editing their classmate’s work, even when 
they knew it was unacceptable. The students felt territorial about their work and respected the boundaries of 
other’s work when presenting their content in the wiki. The wiki was not used as a collaborative writing 
tool. These findings do not support the literature that wiki encourages team writing and editing. 

Phase 2: Adaptation (A Crisis of Relationship) 

Communication After the first wiki article was submitted and the work was found to be of poor quality, 
the instructor provided more guided instruction on how to co-write in wiki, including using the discussion 
board for communication. New teams were established and a new article was required. Participants began 
to communicate with their team members more effectively. The discussion board feature in wiki was 
adopted by the teams as well as face-to-face meetings. Participant 1 stated that appropriate communication 
with the discussion board in wiki increased her comfort level with her peers. She said (Excerpt 10) “I think 
that not being familiar with what to do was really what came at the beginning, not being comfortable. Once 
we used the discussion board on wiki to communicate, we all got on the same page. I think basically my 
comfort level [with using wiki] made a big difference.” Participant 6 described how communication took 
place in her group and how it affected the relationship among team members (Excerpt 11). “We had to be 
sure that we had at least two meetings for the team. First of all, we had to work well with each other. It's 
their timing and my timing. It's like any group project, how to fit and how to work well with the team to get 
a good product.” 
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The first article gave students experience in working in teams and working in wiki. The second article 
allowed students to practice communication skills, both face-to-face and using the wiki discussion board. It 
is argued that such communication helped enhance the relationships among the team members. The 
improved communication positively affected students’ writing quality. 

Writing Students moved from experiencing high anxiety and uncertainty in the first article to improved 
communication and relationships in the second article. Participant 2 explained, (Excerpt 12) “at first it was 
hard to get it. As I’ve gotten better at it, I like it more.” 

When asked about territorial boundaries when using the wiki as a collaborative writing tool, participant 
1 said (Excerpt 13) “if I changed a whole bunch [of text on the article], I posted something about it on 
discussion board. If they didn’t like it, they could change it. It didn’t matter really.” Participant 1 continued 
to explain (Excerpt 14), “I think you are more inclined to edit everybody’s work on the wikis. If s just there 
on the screen so I think the project would be a little bit better quality when it was in the wiki [because of 
open editing].” Participant 6 described the pleasant connection when seeing wiki projects when searching 
Google. She said (Excerpt 15), “now, when I go to any search on Google and find wikis, I feel that is 
something I know. So it is nice for me. It is always nice to know more.” A better writing experience with 
wiki emerged during the second article assignment due to increased skill in using wiki and more explicit 
expectations regarding the quality of the articles. 

Collaboration. Participants continued to test and adjusted their relationship with their peers when 
working on the second article. In this manner, they became more familiar with using wiki and the 
collaborative writing process. They reported an improved experience with communication, writing, and 
collaboration. Participant 4 said (Excerpt 16), “the first time, I had a hard time. After that I got some 
information from my team members, I learned and I started to like [using wiki].” Participant 1 said 
(Excerpt 17) “not understanding the wiki, we wanted to get everything situated before we posted it. I guess 
it’s just the mentality that when you post it on there and that's the final product, so you get it done. As we've 
learned more about wikis, we're able to do everything in wikis.” These comments indicate that when 
communication channels open, writing is less stressful as collaboration was improved by knowing how to 
use wiki more effectively. 

Phase 3: Collaboration (A Resolution of Crisis) 

Communication. During the course of the semester students were asked to write four wiki articles, two 
solo and two in teams. The researchers documented that the participants’ communication moved from 
minimal, either via e-mail or discussion board on wiki to maximum communication via the wiki discussion 
board and face-to-face meetings. Once students learned how to use the wiki, how to edit and co-write, and 
how to communicate with each other more effectively they enjoyed using the wiki to complete the course 
assignments. Participant 4 said (Excerpt 18), “we don’t spend all the time talking, but totally collaborative 
writing. We can exchange our writing through e-mail or via wiki. We have talked a lot on the discussion 
board about what we are going to do, what our views are, and what categories we want to write our stuff in. 
I think that’s collaborative.” 

Participants 6 and 5 compared communicating via the discussion board in wikis with e-mail and 
Blackboard® discussion board. Participant 6 said (Excerpt 19), “we used the discussion board in wiki, it 
was like we are online and we can do it faster than email.” Participant 5 said (Excerpt 20), “comparing with 
wiki, Blackboard®, the course management system, just has the discussion function. If s just like talking in 
class. You can’t really go back and change your answers. With wikis, however, you can keep adding to it.” 

As identified by the participants, communicating with the discussion board on wiki became not only an 
addition to face-to-face meetings, but also an integral part to the group assignments. 

Writing As a result of smoother communications, participants resolved the crisis of authorship and 
relationship, which was observed in the initial phases. Two participants summarized their writing 
experience from the first project to the last. Participant 6 said (Excerpt 21), “after doing 5 chapters in Wiki, 
I'm confident to add and write [in wikis].” Participant 3 said (Excerpt 22), “With the first article, it really 
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intimidated me so I did everything outside of it and just cut and paste. The second article I got more in the 
wiki and started typing in the actual wiki. I did a lot more in the third article. So it was just a matter of 
getting comfortable using the system, a big thing that I think affects whether you like it or not. I’ve enjoyed 
it as I’ve learned more.” 

At the end of the course participants were asked if they felt comfortable editing others’ work. 
Participant 1 said (Excerpt 23), “editing has become interesting to do. Editing someone else’s work was not 
hard in the wiki. I didn’t feel it was another’s work; it was just a web page that I felt I could add something 
to.” The participants indicated that they felt confident about writing and editing in wiki. To this end, wiki 
helped them gain a deeper understanding of the article topic. 

Collaboration Given the promise of wikis for open editing and collaborative learning, the researchers 
continued to investigate whether the students took full advantage of the wiki to learn personally and 
collaboratively. Participant 3 said (Excerpt 24), “I think that it’s really better than in class because it 
actually forces you to choose a research topic. I think I learned a lot better because you are actually forced 
to go out there and search for materials and then by letting your classmates know via wikis.” Participant 6 
compared the unique collaborating experience in wiki with other tools such as encyclopedias. She said 
(Excerpt 25), “the idea of cooperation in wiki allows everybody else to read it online, and they are welcome 
to edit it. No other encyclopedias have this feature.” 

Two participants pointed out that group collaboration helped generate ideas and enhance creativity. In 
addition, it was mentioned that different viewpoints and backgrounds were beneficial for the learning. 
Participant 1 said (Excerpt 26), “in the group you're going to have more ideas because you have more 
people, different viewpoints, and different backgrounds. I think I tend to be less creative working on a 
project individually. I don't spend as much time generating ideas. I just go with what I know.” Participant 5 
said (Excerpt 27) “I thought it was great to see people from different backgrounds. It added new ideas.” 

The results suggest that using wiki as a collaborative tool became more productive and constructive for 
the participants over time as they learned how to use wiki and the elements of a high quality article. The 
findings enrich discussions in the literature that wikis can be used to move learners toward a state of 
communal constructivism, where participants are not simply engaged in developing their own information 
but actively involved in creating knowledge that will benefit others (Holmes, et al., 2001). If the initial 
phases of using wiki as a collaborative tool are underrepresented in the literature, the last phase began to 
unearth the promise of writing in an open environment constructively. 

Discussions 

This study reported that when writing and editing wiki articles collaboratively, students experienced a 
crisis of authority and feared editing their peer’s work, even when the same grade was assigned to every 
member of the team, regardless of individual effort. This situation was especially evident during the initial 
phase of creating and editing a wiki article. 

At first, the participants were not comfortable communicating and writing in wiki. In particular, they 
felt uncomfortable editing their peers’ articles in that they believed their peers had devoted a tremendous 
amount of time and effort on a particular topic while they did not. They experienced a crisis of authority. 
They indicated that until they did their own reading and became very familiar with the material, they 
should not edit other’s work to ensure individual accountability of their editing. As indicated by the 
students, this crisis of authority in the first phase directly resulted from not knowing the functionality of the 
wiki tools and a lacking of effective communication among team members. Not surprisingly, little 
collaboration occurred in phase 1 ; adjustment was dominant. 

In the second phase, as the participants overcame the functionality of wiki, communication increased. 
Free editing behavior increased but it was not common. In fact, the participants were not comfortable 
editing their peers’ articles before they knew their team members’ working styles, reaction to 
interdependence, and attitudes on territorial limits. They experienced a crisis of relationship. In this phase, 
communication was improved, collaborative writing in wikis was visible, and collaborative learning 
emerged. 
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In the third phase, once trust and rapport were established within the student teams, the frequency of 
communication and co-writing increased. Positive interdependence occurred rapidly. Students experienced 
a resolution of crisis. Furthermore, the participants indicated that they usually would consult with the 
original author of the wiki chapters to ensure that the changes were pertinent and value-added, add a crisis 
of negotiation. At this phase, they positively viewed writing in wikis as pair-share, peer-teaching, and peer- 
collaboration, which aligns with the promise that wikis hold for enhancing collaborative learning. 

In summary, students’ communication and writing changed over the life of the course. The idealistic 
hypotheses that wiki work is naturally beneficial and contributes to collaborative learning did not occur. 
Instead, students worked through the crisis of authority and relationship, to resolution through negotiation. 
Such changes were influenced by their estimation of territorial boundaries and interpersonal interactions 
while working in small teams. The following session reflects on the design and implementation practices of 
using wikis in this course. 



Implications for Design and Implementation 

Wikis are a new computer-mediated communication tool that holds much promise for collaborative 
writing projects in the educational setting. Even though students may have been exposed to other tools such 
as email, blogs, course management systems, and e-portfolio, we found that they experienced a steep 
learning curve and did work effectively on the first wiki article assignment. The learning curve included 
technical operation of the software as well as knowing how to truly write collaboratively (Brandon & 
Hollingshead, 1999). Our students were anxious and uncertain about editing others’ writing initially, and 
required a paradigm shift to overcome a crisis of authority. 

Consistent with the literature on using wikis (McKay & Headley, 2007; Qian, 2007), we suggest that 
the instructors design a practice article at the beginning of the course to teach students how to use the 
software and to encourage re-writing others’ text. Many wikis contain a sandbox, but further coaching may 
be necessary to resolve the crisis of authority among groups who have been long taught to not interfere 
with other’s work. A practice article will not only allow students to become familiar with wiki 
functionality, but will model the practice of interacting with peers on the wiki. Instructors are also 
encouraged to model collaborate writing to prompt students’ critical thinking and decision-making skills 
(Engstrom & Jewett, 2005). 

Repeated wiki assignments were also necessary to obtain the benefits of collaborative writing in wiki. 
This course required five articles, 4 original (2 solo, 2 in teams), and 1 edited. The benefits of using wiki as 
a collaborative writing tool were not realized until the third or fourth article assignment. It is recommended 
that instructors use several wiki assignments to reinforce the skills of collaborative writing infused with 
feedback on quality expectations of the articles. 

As more educators explore and adopt the use of Wikis, we hope that this study lays the groundwork for 
further study on team editing behavior to advance the artifact evaluation of human interactions within a 
wiki environment, other than its technical interaction. As Perkins, a pioneering thinker about computers as 
learning tools, concluded, the importance of innovative technologies is not “what you can do that you could 
not before,” rather, the important question to ask is “what difference will computers really make” to a 
person’s higher-order thinking skills such as decision making, reflection, reasoning, problem solving, and 
peer collaboration (Perkins, 1986, p. 1 1). 
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Abstract 

This study investigates content rating and filtering products and models in distance learning. Three distance 
learning content rating and filtering systems (time-based, space-based and functional factor-based) are described at 
the beginning. Then different distance learning content rating and filtering products are examined and categorized 
into the functional factor-based system. Finally six popular distance learning content rating and filtering models are 
discussed and compared in detail. 

Keywords: distance learning, rating, filtering, products, models 

Introduction 

Nowadays the popularity of harmful content on the Internet has become one of the most serious global 
concerns in distance learning systems. In order to deal with this challenging problem, different kinds of distance 
learning content rating and filtering products and models have been developed. 

The purpose of the study is to examine the following questions: 1. How to understand distance learning 
content rating and filtering systems? 2. How to compare and categorize different kinds of distance learning content 
rating and filtering products? 3. What are the advantages and disadvantages of the popular distance learning content 
rating and filtering models? 

The contribution of this study to the literature is twofold. First, different distance learning content rating 
and filtering products are examined and categorized to exhibit a complete concept map in this issue. Second, the 
popular distance learning content rating and filtering models have been discussed in detail to figure out their 
advantages and disadvantages, the useful reference for future improvement. 

Three Distance Learning Content Rating and Filtering Systems 

Usually there are three distance learning content rating and filtering systems (time-based, space-based and 
functional factor-based. The time-based system (Figure 1) reveals the process that distance learning content should 
be labeled with rating result before it can be filtered to learners. Generally both labeling and filtering are based on 
the same rating standard. 




Figure 1 The Time-based System 
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Second, the space-based system (Figure 2) describes distance learning content rating and filtering in 
broader scope. According to the spaced-based system, distance learning content rating and filtering is not just a 
technological issue. The other important parts include legal and social regulations, and information literacy 
education. The three parts interact with each other and contribute to the system in different ways. 




Third, the functional factor-based system (Table 1) gives more detail in technical analysis, which consists 
of rating terminology, rating entity, rating establishment, rating transmission and filtering implementation. 

The rating terminology includes descriptive ratings terminology and evaluative rating terminology. For 
example, “too much violence” belongs to descriptive ratings terminology, while “unsuitable for 6-year-old or 
younger children” is evaluative rating terminology. Generally, descriptive ratings terminology is simple and easily 
used but it lacks detailed information about the content, while evaluative rating terminology provides content 
description but it is not as user-friendly as descriptive ratings terminology. 

The rating entity can be content developers, content packagers, third-parties or learners. It is possible for 
content developers to rate their own content products based on rating standards, such as ICRA and SafeSurf, while 
content packagers, such as portal websites and databases, can also possibly rate and edit the content they buy or 
search from content developers. Third-parties include governments, NGO (Non-Governmental Organizations) and 
companies. For example, many companies, such as Cyber Snoop, Click & Browse Jr., AME, Cyber Patrol, 

SurfW atch (the name now has been changed to SurfControl), invite parents, instructors and IT experts to set up 
working groups to rating online content. Sometimes learners are also responsible for the rating. 

The rating establishment can be described as authoritative and democratic. The authoritative rating 
establishment means the rating can only be implemented by authorized institutes, while the democratic rating 
establishment means the rating implementation should be shared by all the stakeholders, including voting, 
investigation and auto-rating based on open standards, such as Net Shepherd. 

The rating transmission can be keywords or meta-information. Meta- information is an abstract description 
about a webpage, a picture or a movie. For example, W3C develops PICS (Platform for Internet Content Selection), 
a HTML rating and labeling specification, so a PICS rating result of a webpage can be embedded in the HTML 
heading of the webpage. When client software identifies the PICS label in the webpage, it can decide if the webpage 
should be filtered. 

The filtering implementation is more complex. In relation to the filtering location, the filtering 
implementation can be upstream filtering or downstream filtering. In line with the filtering method, the filtering 
implementation can be “hard” filtering, which means content has been rating-labeled and filtered without learners’ 
consent, or “soft” filtering, which informs learners about content rating-label and it is learners who decide if the 
content should be filtered or not, such as Yahooligans!, American Library Association's "Great Sites", CyberYES 
list, The Internet Kids & Family Yellow Pages. 

From the analysis of the above three distance learning content rating and filtering systems, it can be found 
that there are too many complex factors in the issue, so emphasizing or deemphasizing any of these factors will 
result in low efficiency of the whole system. 
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Table 1 The Functional Factor-based System 



Rating terminology 



descriptive ratings terminology 



evaluative rating terminology 



content provider 



Rating entity 



content packer 



third party 



learner 



authoritative 



Rating establishment 



democratic 



voting 



investigation 



auto-rating based on open standards 



Rating transmission 



content name 



content meta- information 





location 


upstream 




downstream 






“hard” 


destroy 


Filtering implementation 




screen 




method 




recommendation 






“soft” 


supervision 








warning 



Six Distance Learning Content Rating and Filtering Models 

There are six popular distance learning content rating and filtering models. The first one is the keyword- 
based model. For example, we can use the keyword “too much violence” to rate a picture. This model is simple, 
user-friendly, highly flexible but less accurate since it is difficult for this model to figure out the difference between 
sex business and sex education. The second model is called greenlist and redlist model. Here greenlist means the 
recommended website list, while redlist means blocked website list. Both of the website lists are set up manually, so 
the lists can only cover very few websites on the Internet and they need a long period to update. The walled garden 
model is the third one, which means a restricted online community where content providers should be responsible 
for its content. It’s very safe for learners but sometimes learners have to pay the service fee and they can only access 
the content within the online community. The fourth one, called the Internet top domain name model, which is 
proposed by ICANN (Internet Corporation for Assigned Names and Numbers) to help learners figure out adult 
websites by the Internet top domain name (such as .sex, .red and .xxx) of the websites, but many adult websites 
refuse to adopt this plan. The fifth one, the learner authentication model, asks learners to provide age information to 
an authentication server before they can access content, but many content providers don’t take it into consideration. 
The last model, called MPLRM (multi-party labeling & rating model), requires content providers to label content 
based on rating standard firstly, then client software can identify the rating label in the meta-information of content, 
but the globally accpeted rating standard is not available now. The six distance learning content rating and filtering 
models are compared at Table 2 and Figure 3, which indicate that advantages and disadvantages are available in all 
the models. However, MPLRM is generally the best choice since its flexibility-accuracy area is the biggest 
according to Figure 3. In other words, MPLRM keeps a good balance between the flexibility and accuracy. 
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Table 2 Advantages and Disadvantages of Six Models 



Model 


Advantage 


Disadvantage 


Keyword-based model 


highly flexible 


lowly accurate 


Greenlist and redlist model 


highly accurate 


little Internet coverage 
low updating speed 


Walled garden model 


highly accurate 


little Internet coverage 
service fee 


Internet top domain name model 


highly accurate 


depending on content providers 


Learner authentication model 


highly accurate 


depending on content providers 


Multi-party labeling & rating model 


accurate and flexible 


depending on content providers 




Figure 3 Advantages and Disadvantages of Six Models 
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Abstract 

This study examines the strengths and weaknesses of two currently popular kinds of e-learning content 
rating (age-based and content-based) standards to construct a new e-leaming content rating standard developing 
strategy called Double Face Model (DFM), which combines age-based rating and content-based rating at the same 
time but in different dimensionalities. 
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Introduction 

The Internet has been widely accepted as the important learning platform for learners in recent decades. 
Despite the educational benefits of the Internet, however, there are risks associated with its use, such as exposure to 
inappropriate material, physical danger and unwanted advertising towards e-leamers. Nowadays the harmful content 
on the Internet has become one of the most serious global concerns in e-learning systems. Therefore, many 
governments and international professional organizations, such as Internet Content Rating Association (ICRA, 

2003), have published online content rating standards to deal with this challenging problem. 

The purpose of the study is to examine the following questions: 1. What are the strengths and weaknesses 
of two currently popular kinds of e-learning content rating (age-based and content-based) standards? 2. Why doesn’t 
the high content accuracy of e-leaming content rating standards equal users’ highly positive attitude to these 
standards? 3. Is it possible for us to design a new e-leaming content rating standard developing strategy to change 
the embarrassing situation? 

The contribution of this study to the literature is twofold. First, the strengths and weaknesses of two popular 
kinds of e-leaming content rating (age-based or content-based) standards are not typically examined by the current 
literature. This study contributes to the existing literature by the comparison of two kinds of e-leaming content 
rating standards, which reveals the high content accuracy of e-leaming content rating standards doesn’t equal users’ 
highly positive attitude to these standards. Second, this study constmcts a new e-leaming content rating standard 
developing strategy called Double Face Model (DFM), which combines age-based rating and content-based rating at 
the same time but in different dimensionalities to deal with this problem. 

Problem 

E-leaming content rating standards in this study is defined as the specifications for judging the relative 
quality of e-learning content for e-leamers. According to e-leaming content rating standards, the e-leaming content 
is labeled with specific rating result, which helps e-learners judge if they should access the content by indicating the 
relative quality of the e-leaming content. 

Currently so few content rating standards are specifically designed for e-leaming and most e-leaming 
systems apply the common content rating standards, which are developed in two different ways: age-based or 
content-based. The age-based content rating standards indicate the relative quality of content by age-related labels, 
such as “suitable for children ages 3 and older” (ESRB, 2003), while the content-based content rating standards 
indicate the relative quality of content by key- word description, such as “explicit or cmde language” (RASCi, 2003). 

In the pre-Internet age, people usually focus on content rating in broadcasting, television, movie and 
electronic game, so the rating standards in this period are simply based on age without flexible mles (Table 1). That 
is to say, according to their age, people can only choose to accept or reject the content, but they can’t choose when 
and where to accept or reject the content. 
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Table 1 Age-based Content Rating Standards 



Age (year) 


1 


2 


3 


4 


5 


6 


7 


8 


9 


10 


11 


12 


13 


14 


15 


16 


17 


18 


19 


20 


MPA 1 










































MPAA 2 










































ESRB 3 










































B.C. 4 










































U.K. 5 










































H.K. 6 











































1 : MPA: the abbreviation for rating standard of Motion Picture Association. 

2 : MPAA: the abbreviation for rating standard of Motion Picture Association of America. 

3 : ESRB: the abbreviation for rating standard of Entertainment Software Rating Board. 

4 : B.C: the abbreviation for rating standard of Province of British Columbia in Canada. 

5 : U.K: the abbreviation for rating standard of United Kingdom. 

6 : H.K: the abbreviation for rating standard of Hong Kong S.A.R. of P.R.China. 

With the rapid development of information technology, nowadays people focus on developing new content 
rating standards for the Internet. Although content rating standards based on age still prevail in the pre-Internet age, 
new standards grounded on content are needed at this time. On the one hand, there are two new characters of 
harmful content on the Internet: First of all, it spreads so fast that people can get such content in seconds from the 
other side of the Earth by simply clicking a mouse button. Second, there is no national border, and there are no 
resting hours either, therefore people can search whatever they want in 24 hours globally, including harmful content. 
These new characters make it impossible to insist on content rating standard development strategy for pre-Internet 
age. On the other hand, supported by the emerging information technologies, people are beginning to adopt new 
strategies based on content to develop more flexible rating standards, which are different from the index of 
prohibited books prevailed in middle age or rating standards based on age for the pre-Internet age. More information 
can be gotten from Table 2, which compares several well-known standards in the Internet age, including Cyber 
Patrol, SurfWatch, RSACi and CYBERsitter. 



Table 2 Content-based Content Rating Standards 



Option 


Cyber Patrol 


SafeSurf 


RSACi 


CYBERsitter 


ICRA 


sex/nudity 


• 


• 


• 


• 


• 


violence 


• 


• 


• 


• 


• 


profanity 


• 


• 


• 




• 


satanism 


• 






• 




discrimination 


• 


• 


• 


• 




dmg 


• 


• 






• 


tobacco/ bibulosity 


• 








• 


gambling 


• 


• 






• 


others 


• 






• 





Comparing the two kinds of content rating standard, we can find the following differences between them 
(Table 3): First, age-based standards are amazingly similar despite their different social, economic and cultural 
background, but content-based standards have their own specific rating characteristics and they have often been 
changed. Second, age-based standards are more stable, while content-based standards are more flexible. Third, E- 
leaming content providers prefer content-based standards because they are familiar with the content, while e- 
leamers often choose age-based standards since age is a much simpler indicator for them compared with content 
description. Neither of the two kinds of rating standards can work satisfyingly for both e-leaming content providers 
and e-leamers at the same time. That’s the reason why the high content accuracy of e-leaming content rating 
standards doesn’t equal users’ highly positive attitude to these standards. 
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Table 3 Comparison of Age-based and Content-based Content Rating Standards 



Standards based on age 


Standards based on content 


simple 


complex 


similar 


different 


stable 


flexible 


mainly for e-learning content providers 


mainly for e-leamers 



Method 

Double-face model (DFM) combining the strength of age-based and content-based e-leaming content rating 
standards is proposed in this study as a possible solution to deal with this dilemma, which includes two 
dimensionalities: dimensionality A and dimensionality B. That is to say, there are two rating results for the same e- 
leaming content. The rating result from dimensionality A aims at e-leaming content providers, while the rating 
result from dimensionality B is for e-learners. DFM provides a mapping mechanism for combining dimensionality A 
with dimensionality A, so two rating results based on dimensionality A and dimensionality B respectively can be 
converted into each other. Furthermore, the transformation is “transparent” to e-learning content providers and e- 
leamers. (Figure 1) 




Figure 1 Double-face Model 

Dimensionality A, which is based on content, provides two content categories: recommended and blocked. 
Each content category is divided into several content options, and each content option can be further divided into 
several grades (Table 4). 



Table 4 Dimensionality A 



Label 


Option 


Sub-option 


Grade 


A 


recommended 


excellent educational content 


0-4 


B 


educational content suitable for special age 


0-4 


c 


undefined content 


0-4 


D 


blocked 


bad language 


0-4 


E 


tobacco, bibulosity, dmg 


0-4 


F 


mmor 


0-4 


G 


terror 


0-4 


H 


violence 


0-4 


I 


wizardry 


0-4 


J 


gambling 


0-4 


K 


militarism, extremism, terrorism 


0-4 


L 


discrimination 


0-4 
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M 




nudity, sex 


0-4 


N 


Satanism, cult 


0-4 


0 


racialism 


0-4 


p 


endangering the national security 


0-4 


Q 


endangering the Internet safety 


0-4 


R 


other illegal issues 


0-4 


s 


undefined content 


0-4 



The rules of description based on dimensionality A are as follows: According to given e-leaming content, 
we can choose grades labeled from 0 to 4 for each sub-option. 0 means null for this sub-option and 4 means the 
strongest. Each sub-option with its grade is divided by For example, if there is a webpage which propagandizes 
violence, its rating result based on dimensionality A is: A0-B0-C0-D0-E0-F0-G0-E14-I0-J0-K0-L0-M0-N0-O0-P0- 
Q0-R0-S0, which complies with W3C PICS. 

Dimensionality B provides eleven sub-options (Table 5), and each sub-option can map to some sub-options 
in dimensionality A. (Table 6-11) 



Table 5 Dimensionality B 



Label 


Definition 


Blc-All 


Unsuitable for all e-leamers 


Blc-18older 


Only suitable for 18 year’s old or older e-leamers 


Blc-12older 


Only suitable for 12 year’s old or older e-leamers 


Blc-Un 


Block reserved 


Rec-All 


Recommended for all e-leamers 


Rec-1517 


Recommended for 15-17 year’s old e-leamers 


Rec-1214 


Recommended for 12-14 year’s old e-leamers 


Rec-711 


Recommended for 7-11 year’s old e-leamers 


Rec-6younger 


Recommended for 6 year’s old or younger e-leamers 


Rec-Un 


Recommendation reserved 


Default 


not recommended 



Table 6 The Mapping between Dimensionality A and Dimensionality B 
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Table 7 Rec-All 



Grade 


excellence 

educational 

content 


educational content 
suitable for special age 


undefined 

content 


4 








3 






2 






1 








0 








Table 8 Rec-6younger 


Grade 


excellence 

educational 

content 


educational content 
suitable for special age 


undefined 

content 



4 




Table 9 Rec-711 



Grade 


excellence 

educational 

content 


educational content 
suitable for special age 


undefined 

content 


4 


■M 






3 


1 Hi 


1 




2 








1 


1 M 






0 


| 







Table 10 Rec-1214 



Grade 


excellence , . , , . 

, . t educational content 

educational t ~ , 

suitable for special age 

content 


undefined 

content 


4 






3 






2 




1 






0 







Table 11 Rec-1517 
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DFM provides a mapping system to combine dimensionality A and dimensionality B (Figure 2). For 
example, there is a webpage which propagandizes violence. According to dimensionality A, the rating result that e- 
leaming content providers get is: 

A0-B0-C0-D0-E0-F0-G0-H4-I0-J0-K0-L0-M0-N0-O0-P0-Q0-R0-S0 

Then the e-leaming content provider sends the result to RFC (rating & filtering center) for registration. 
RFC is an abstract concept, which can be implemented as software, web service, etc. RFC finishes the mapping 
between dimensionality A and dimensionality B and gets the mapping result: Blc-All. That is to say, it’s unsuitable 
for all e-leamers. At the same time, an e-leamer adjusts age setting on his client software, which sends the e- 
leamer’s age information to RFC. Finally, RFC finds that the webpage is unsuitable for this e-leamer according to 
the mapping result, so RFC blocks the webpage automatically. 

mapping 



H4 % Unsuitable for all e-leamers 




e-leaming content provider 



e-learner 



Figure 2 The Mapping System between Dimensionality A and Dimensionality B 



Conclusions 

First, DFM provides the transparent mapping mechanism between dimensionality A and dimensionality B, 
so the two rating dimensionalities can be converted into each other conveniently. Second, DFM provides the 
“recommended” content options, which are not available in other e-learning content rating standards. Finally, there 
are reserved content rating options in DFM such as “content suitable for special age” and “undefined content”, 
purposely designed to satisfy some e-learners’ specific needs such as religious or local culture emphasis. 
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Abstract 

This study examines Shanghai’s high school students’ cognition and attitude towards online pornography. 
In order to obtain a comprehensive perspective of this issue, three comparisons have been conducted in this study: 
between teachers and students at Shanghai high schools, between male students and female students at Shanghai 
high schools, and between Shanghai’s high school students and Taiwan Province’s high school students. 

Keywords: students, cognition, attitude, online pornography 

Introduction 

The survey from CCYL (China Community Youth League, 2002) indicates the Internet has become an 
important learning platform for Chinese young learners in recent decades. Despite the educational benefits of the 
Internet, however, there are risks associated with its use, such as online pornography towards e-learners, which is 
demonstrated by many other related surveys. For example, the survey from Yesite.com (2002) indicates that more 
than 60% Chinese young people access online pornography. Another investigation from Sina.com (2002) indicates 
that more than half of two million Chinese students access online pornography. It’s widely recognized that the 
overwhelming pornography on the Internet has become one of the most serious social concerns in China nowadays. 

The purpose of the study is to examine the following questions: 1. What are the differences of cognition 
and attitude towards online pornography between teachers and students at Shanghai high schools? 2. What are the 
differences of cognition and attitude towards online pornography between male students and female students at 
Shanghai high schools? 3. What are the differences of cognition and attitude towards online pornography between 
Shanghai’s high school students and Taiwan Province’s high school students? 

The contribution of this study to the literature is listed as follows. First, the differences of cognition and 
attitude towards online pornography between teachers and students at Shanghai high schools provide reference for 
high school teachers to set up effective conversation with students in this issue. Second, the differences of cognition 
and attitude towards online pornography between male students and female students at Shanghai high schools 
demonstrate we should take the gender factor into consideration when designing instruction related to this issue. 
Third, the differences of cognition and attitude towards online pornography between Shanghai’s high school 
students and Taiwan Province’s high school students reveal the social and cultural influence in this issue. 

This study designed a questionnaire about cognition and attitude towards online pornography for teachers 
and students in Shanghai high schools. In 2003, this study collected a total of 1,206 qualified questionnaires from 
1,245 Shanghai high school students in and a total of 34 qualified questionnaires from 40 Shanghai high school 
teachers. Based on the questionnaire survey results and the data from the high school student online survey in 
Taiwan Province (2002) about their cognition and attitude towards online pornography, this study completed the 
following comparisons. 

The Differences between Teachers and Students at Shanghai High Schools 

First, the students and the teachers access the Internet at different locations, a majority of the students often 
surf the Internet at Internet cafes (54%) , while many of the teachers choose to access the Internet at school (56%) or 
at home (44%). None of the teachers in the survey go to Internet cafes. 

Second, the average time (2.6 hours) which the students spend on the Internet every day is much longer 
than the teachers' (0.9 hours), which is a possible reason why students have much more Internet experience and 
skills compared with the teachers. 
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Third, 62% of the students access online pornography, intentionally or unintentionally, but the number for 
teachers is only 4%. 44% of the teachers are not familiar with online pornography. 49% of the teachers just hear it 
from others and 44% don’t realize pornography is possibly accessible on the Internet. 

Fourth, 60% of the students who access online pornography find it from search engines or friends, while 
41% of the teachers who access online pornography find it from friends. 

Fifth, 78% of the students and 44% of the teachers think online pornography is most popular on web pages. 
At the same time, 36% of the students and 47% of the teachers think it can also be found easily at online chat rooms. 
It demonstrates nowadays popular information technologies used at their schools doesn’t provide effective protection 
against online pornography. In addition, 84% of the students and 62% of the teachers think people can access online 
pornography most easily at Internet cafes. 

Sixth, 64% of the students and 53% of the teachers think “accessing online pornography doesn’t matter” 
and even 42% of the students feel “curious” about online pornography. 

Seventh, 76% of the students and 88% of the teachers have never received the concept about media 
education. Only 30% of the students got limited media guidance from their parents before. However, the parents 
have little information technology skills compared with their children, which cannot guarantee the effectiveness of 
their guidance. 

Finally, the students and the teachers have different answers for the question about how to deal with online 
pornography. 49% of the students believe the self-regulation is the best way to deal with the problem, while 45% of 
the teachers think legal enforcement is more important. Furthermore, 64% of the students and 82% of the teachers 
are not familiar with online content rating standards or products, but both the students (81%) and the teachers (79%) 
think information literacy education in high schools should be improved. 

The Differences between Male Students and Female Students at Shanghai High Schools 

First, 71% of the male students and 47% of the female students access online pornography. They are 
interested in pom articles, pom photos and pom chat rooms, but what they are most interested in is pom photos. 
Among them, 69% of the male students and 67% of the female students choose to access online pornography at 
Internet cafe, and the time is usually from 6pm to 12am. 

Second, in response to the question why access online pornography, 48% of the male students choose 
“curiosity” and 16% of the male students think “if I don’t access it, I will be laughed by my friends”. However, 85% 
of the female students choose “other reasons”. 

Third, the male and female students access online pornography in many different ways. 62% of male 
students and 33% of female students in the survey find online pornography by search engines, while 33% of male 
students and 22% of female students find it from friends or classmates. 

Fourth, among the students who access online pornography, 95% of male students visited it before and 
28% of male students often visit it. For female students, the percentages are 60% and 48% respectively. 

Fifth, 55% of the male students access less than 5 pornography websites and 15% of the male students 
access more than 20 pornography websites. However, only 34% of the female students access less than 5 
pornography websites and 3% of the female students access more than 20 pornography websites. 

Sixth, among the students who visited online pornography, 69% of male students and 67% of female 
students choose Internet cafes as the first choice to visit online pornography. Furthermore, 26% of the male students 
also choose home as their second choice, while 31% of the female students also choose friends’ house as their 
second choice. Most of the male and the female students choose to visit online pornography from 6p to 12a. 

Finally, more than half of the male students love to discuss online pornography with their friends, but most 
of the female students don’t tell anyone, even their close friends. Moreover, both male students (73%) and female 
students (49%) think “accessing online pornography doesn’t matter”. 

The Differences between Shanghai’s High School Students and Taiwan Province’s High School Students 

First, 85% of the high school students in Taiwan access online pornography compared with 83% of the 
high school students in Shanghai. 41% of the high school students in Taiwan access online pornography compared 
with 62% of the high school students in Shanghai. 

Second, the high school students’ online pornography visiting frequencies are different. Among the high 
school students who visited online pornography, 23% of Shanghai students often visit, 41% of Shanghai students 
sometimes visit and 36% of Shanghai students rarely visit; while 13% of Taiwan students often visit, 31% of Taiwan 
students sometimes visit and 56% of Taiwan students rarely visit. 
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Third, 62% of the high school students in Taiwan choose to visit online pornography at home while 68% of 
the high school students in Shanghai choose to visit it at Internet cafe. 

Finally, 73% of the high school students in Taiwan feel “curious” about online pornography compared with 
42% of the high school students in Shanghai. 



Conclusions 

First of all, the percentages of both of male and female high school students who access online 
pornography in Shanghai are very high. Therefore, it is important to implement necessary guidance at Shanghai high 
schools. Second, the high school students and teachers in Shanghai visit the Internet at different locations and 
different times, so the teachers cannot provide effective guidance to the students. Third, the high school students in 
Shanghai access online pornography for varied reasons, so the individual guidance plans are necessary for different 
students. Fourth, the Shanghai high school teachers know less about online pornography than the high school 
students. This embarrassing situation should be changed by new learning modules in the teacher’s professional 
training program. Finally, the differences between Shanghai high school students and Taiwan high school students 
reveal the social and cultural influence in this issue, which indicates more related researches could be done in the 
future. 
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Abstract 

Several studies in traditional classroom settings examined instructor immediacy (the measure of the 
psychological distance which an instructor puts between himself and his students) and reported that 
instructor verbal and nonverbal immediacy behaviors are associated with learning outcomes, satisfaction, 
and motivation. The purpose of this study was to examine the effect of instructor immediacy behaviors on 
student perception of instructor immediacy and social presence (the degree to which a person is perceived 
as “real” in mediated communication) in two online, computer conferencing environments: (a) video and 
audio with text chat and (b) audio with text chat. Using an experimental design the researchers tested the 
effects of instructor immediacy behaviors (high vs. low) and delivery modality (audio vs. video) on student 
perception of instructor immediacy, perception of social presence, and learning outcomes measured by 
immediate posttest scores. The results yielded statistically significant difference in student perception of 
instructor immediacy and social presence among the high-immediacy and the low-immediacy groups. The 
results also showed that there was a significant difference in learning outcomes as indicated by immediate 
posttest scores between students in the high-immediacy audio group and the low-immediacy video group. 

A correlation analysis revealed a significant positive relationship between perceived instructor immediacy 
and perceived instructor social presence. Finally, a regression analysis revealed that instructor immediacy 
significantly predicted social presence. The findings have significant implications regarding the use of 
instructor immediacy behaviors in computer conferencing settings and lay the groundwork for future 
research in this area. 

Currently, the number of students enrolling in courses offered online is increasing at a much faster 
rate of growth than the overall higher education population (Sloan Consortium, 2005) and computer-based 
tools have been widely embraced for online learning. With the prevalence of computers in online distance 
education, there is an increasing need to exploit the differences among the plethora of computer-based 
tools, such as synchronous and asynchronous text chat tools, discussion boards, computer audio and video 
conferencing, and so forth, regarding their impact on the mediated learning transaction and the learning 
outcomes. What is the effect of the behaviors projected by the instructor when using different types of 
online computer-conferencing tools? Unfortunately the central role of the instructor and the impact of his or 
her behaviors have received little attention in the distance learning research. The current study examined 
the effect of immediacy behaviors employed by the online instructor in two computer-based learning 
environments. 



Instructor Immediacy and Social Presence 

Research in conventional classroom settings has examined instructor immediacy, the measure of 
the psychological distance which a communicator puts between himself and the object of his 
communication (Wiener & Mehrabian, 1968). Researchers have identified verbal and nonverbal 
communicative behaviors which may be employed by instructors to reduce student perceptions of 
psychological distance and enhance closeness and interaction. For example, nonverbal behaviors include 
eye contact, body posture, gestures, facial expressions, and vocal qualities (Andersen, 1979; Richmond, 
Gorham, & McCroskey, 1987). Relevant verbal behaviors include using students’ names, feedback, praise, 
and humor (Gorham, 1988). Research on instructor immediacy has shown that when instructors employ 
verbal and nonverbal immediacy behaviors students demonstrate increased learning outcomes, motivation, 
and satisfaction (Andersen, 1979; Christophel, 1990; Gorham, 1988; Gorham & Christophel, 1990; 
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Gorham & Zakahi, 1990; Kearney, Plax, & Wendt-Wasco, 1985; Kelley & Gorham, 1988). While these 
findings received a lot of attention in the communication literature, most of the studies have been 
conducted in traditional face-to-face, non-mediated settings and very few studies have examined instructor 
immediacy in the context of distance education classroom, primarily in the televised classroom (Freitas, 
Myers, & Avtgis, 1998; Guerrero & Miller, 1998; Hackman & Walker, 1990). Consequently, some of the 
verbal and nonverbal behaviors that have been described to enhance instructor immediacy in face-to-face 
interaction might not be feasible or relevant in distance education settings. However, existing research has 
reported notable effects of instructor immediacy on student learning outcomes. Considering the increasing 
number of students enrolled in courses offered through the Internet, there is a noticeable gap when it comes 
to investigating how students perceive instructor immediacy when learning occurs online, through various 
computer conferencing tools. 

While there is little research on immediacy in the context of online learning, several studies in 
distance education and communication have examined the concept of social presence, the degree of 
salience of the other person in a communication transaction (Short, Williams, & Christie, 1976) or as it has 
been widely interpreted, the degree to which a person is perceived as “real” in a mediated communication. 
Researchers investigating social presence suggest that the construct of social presence is closely related to 
the construct of immediacy. Findings in this area of research suggest that student perception of social 
presence can increase student satisfaction and perceived learning outcomes (Gunawardena & Zittle, 1997; 
Picciano, 2002; Richardson & Swan, 2003). However, social presence is partially determined by the 
objective qualities of the medium used in the mediated interaction; thus selecting the appropriate 
communication medium for an instructional instance could affect the student learning outcomes of the 
mediated interaction (Tu & Mclsaac, 2002). 



Methodology 

The purpose of this study was to investigate how students perceive instructor immediacy and 
social presence in computer conferencing sessions. The study focused on two widely employed 
combinations of computer conferencing tools that allow synchronous computer communication: video and 
audio with text chat and audio with text chat. In addition, the study sought to determine whether the use of 
different computer conferencing environments would result in differences in learning outcomes when the 
instructor manipulates the level of immediacy behaviors. Finally, the study examined the relationship 
between perceived instructor immediacy, perceived social presence, and learning outcomes in each of the 
two combinations of computer conferencing environments and instructor immediacy behaviors. 

The study examined the following research questions: 

RQ1 : How does the level of immediacy behaviors projected by the instructor and the computer 
conferencing environment influence perceived instructor immediacy? 

RQ2: How does the level of immediacy behaviors projected by the instructor and the computer 
conferencing environment influence perceived instructor social presence? 

RQ3: How does the level of immediacy behaviors projected by the instructor and the computer 
conferencing environment influence learning outcomes? 

RQ4: Within the context of the different computer conferencing environments — (a) video and 
audio with text chat and (b) audio with text chat — what is the relationship between perceived instructor 
immediacy and perceived instructor social presence? 

RQ5: Within the context of the different computer conferencing environments — (a) video and 
audio with text chat and (b) audio with text chat — what is the relationship between perceived instructor 
immediacy and learning outcomes? 

To explore these issues, the researchers recorded four versions of an online synchronous session to 
reliably manipulate the level of instructor immediacy behaviors while using the two different computer 
conferencing environments (video and audio with text and audio with text). To experimentally compare 
students’ perception of instructor immediacy and social presence in the two computer conferencing 
environments, the instructor engaged students in each of the two environments in a typical lecture 
discussion with identical content and activities, while manipulating the level of verbal and nonverbal 
immediacy behaviors. The high- and low-immediacy conditions were established using existing immediacy 
behaviors derived from immediacy research and the sessions were recorded. The participants for this study 
were sampled from two sections of an undergraduate course in psychology at San Diego State University. 
Combined, the two sections provided a sample of 989 subjects which were randomly assigned to one of 
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four groups and were asked to view a version of the lesson (see Table 1). Of those students, 433 
participated in the study. 



Table 1. Subject Groupings 



Conditions 


Groups 






Video- Audio-Text (VAT) 


Audio-Text (AT) 


High Immediacy 


Group 1: 


Group 2: 


(Hi) 


High Immediacy - Video- Audio-Text 


High Immediacy - Audio-Text 




(Hi-VAT) 


(Hi-AT) 


Low Immediacy 


Group 3 : 


Group 4: 


(Lo) 


Low Immediacy - Video- Audio-Text 


Low Immediacy - Audio-Text 




(Lo-VAT) 


(Lo-AT) 



Findings 

Overall, the findings of this study show that when instructors employ behaviors which can reduce 
psychological distance, students perceive the instructors as more immediate, caring, empathetic, disclosing 
personality, and expressing emotions. Research question one asked how the level of immediacy behaviors 
projected by the instructor and the computer conferencing environment influence perceived instructor 
immediacy. To answer this research question the data were submitted to one-way ANOVA analysis which 
yielded significant results (F( 3, 433) = 91 .912, p = .000). Figure 1 shows that students who viewed the 
high-immediacy sessions (Group 1 and Group 2) indicated higher perception of instructor immediacy than 
students who viewed the low-immediacy sessions (Group 3 and Group 4). Specifically, students assigned to 
Group 1 (Hi-VAT) indicated the highest perception of instructor immediacy, followed by Group 2 (Hi- AT), 
Group 3 (Lo-VAT) and Group 4 (Lo-AT). The findings suggest that video-enabled computer conferencing 
tools can facilitate the projection of more immediacy behaviors; however instructors can project immediacy 
behaviors with the use of audio alone. 




Figure 1. Means plot for instructor immediacy. 

The second research question two asked: How does the level of immediacy behaviors projected by 
the instructor and the computer conferencing environment influence perceived instructor social presence? 
The one-way analysis of variance detected a significant overall F( 3, 433)= 154.337,/? = .000. Figure 2 
shows that Group 1 (Hi-VAT) indicated the highest perception of instructor social presence (M= 36.33, SD 
= 6.018), followed by Group 2 (Hi-AT) (M= 35. 81, SD = 7.006), Group 3 (Lo-VAT) (M= 21.56, SD = 
7.945), and Group 4 (Lo-AT) (M= 20.47, SD = 7.967). These findings are also supported by an open-ended 
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question which asked students to indicate if they perceived the instructor as a real person. Real person was 
defined as a person who is caring, empathetic, disclosing personality, and expressing emotions, in 
accordance to the social presence scale used in this study. More students in Group 1 (98 out of 107 
students) perceived the instructor’s social presence, followed by Group 2 (66 out of 100 students), Group 3 
(64 out of 1 15 students), and Group 4 (56 out of 1 12 students). 




Figure 2. Means plot for instructor social presence. 



Contrast tests showed that students who viewed the high-immediacy sessions indicated 
significantly higher perception of instructor social presence than the students who viewed the low- 
immediacy sessions (p = .000). However, there was no significant difference in perception of instructor 
social presence between students in the two high-immediacy groups (Hi- VAT and Hi- AT). Similarly, there 
was no significant difference in perception of instructor social presence between students in the two low- 
immediacy groups (Lo-VAT and Lo-AT). These findings suggest that the level of instructor immediacy 
projected by the instructor influenced students’ perceptions of instructor social presence; however, the use 
of video or not did not significantly affect whether students perceived the instructor as a real person. 

Similarly, research question four asked if there is a relationship between perceived instructor 
immediacy and perceived instructor social presence. The correlation analysis displayed in Figure 3 showed 
a significant relationship between perceived instructor immediacy and perceived instructor social presence 
(r(433) = .844, p = .000). In addition, a regression equation was developed to see if immediacy is a 
predictor of social presence. Instructor immediacy significantly predicted social presence (F( 1, 433) = 

1 067.567, p = .000). Table 3 shows that the adjusted R squared value was .712. This indicates that 71.2% of 
the variance in social presence can be predicted by perception of instructor immediacy. 



Table 3. Model Summary for Immediacy 



Model 


R 


R Square 


Adjusted R 
Square 


Std. Error of 
the Estimate 


1 


.844(a) 


.712 


.712 


5.624 



a Predictors: (Constant), Immediacy 
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Figure 3. Correlation of social presence and immediacy. 

Research question three asked if the combination of immediacy behaviors projected by the 
instructor and the learning environment resulted in differences in the learning outcomes in the four groups. 
It was hypothesized that students who viewed the high- immediacy sessions (Group 1 and Group 2) would 
indicate higher learning outcomes than the low-immediacy groups, with Group 1 (Hi-VAT) indicating the 
highest learning outcomes. Learning outcomes were measured with an immediate and a delayed posttest. A 
one-way analysis of variance demonstrated a significant difference between the groups on the learning 
outcomes as measured by the immediate posttest scores (F( 3, 433) = 2.736, p = .043) but no significant 
difference was detected on the delayed posttest scores(F(3, 433) = .964, p = .410). Students assigned to 
Group 2 (Hi- AT) achieved the highest test scores (M= 5.40, SD = 1.206), followed by students in Group 4 
(Lo-AT) (M = 5.28, SD = 1.409), Group 1 (Hi-VAT) (. M= 5.22, SD = 1.499), and Group 3 (Lo-VAT) (. M= 
4.87, SD = 1.617). Contrast tests showed that the only significant difference in learning outcomes as 
measured by posttest scores was between Group 2 (Hi- AT) and Group 3 (Lo-VAT) 

(p = .008). Furthermore, research question five asked: Within the context of the different computer 
conferencing environments — (a) video and audio with text chat and (b) audio with text chat — what is the 
relationship between perceived instructor immediacy and learning outcomes? The hypothesis was that there 
is a positive relationship between perceived instructor immediacy and learning outcomes. The correlation 
analyses used to examine research question five did not reveal a relationship between perceived instructor 
immediacy and learning outcomes as indicated by the immediate (r(433) = - .003, p = .945) or delayed 
posttest (r(433) = - .014 ,p = .11 A). 

The last part of the survey asked students to indicate whether they perceived the instructor as a 
real person (caring, empathetic, disclosing personality, and expressing emotions). Most students (98 out of 
107 students) in Group 1 (Hi-VAT) and about two thirds of the students (66 out of 100) in Group 2 
(Hi-AT) indicated that they perceived the instructor as a real person. Significantly fewer students reported 
perceiving the instructor as a real person in Group 3 (Lo-VAT) (64 out of 1 15 students) and Group 4 
(Lo-AT) (56 out of 122 students). Students reported that some of the factors that influenced their perception 
of the instructor as a real person included the instructor encouraging students to ask questions, answering 
questions and providing feedback, using gestures, using examples, calling students by their first name, and 
not using monotone voice. Table 4 summarizes the aspects of the lesson that made students feel like the 
instructor was not real in the four groups. Table 5 presents the categories summarizing the students’ 
opinions about the aspects of the lesson that made the instructor seem real. 
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Table 4. Aspects of Lesson That Made Students Feel Like Instructor Was not Real 



Group 1 


Group 2 


Group 3 


Group 4 


Boring 


Did not move-No video 


Monotone Voice- 


Did not move-No video 


Seemed rehearsed 


Robotic 


Boring 


Monotone Voice- 






Showed no emotion 


Boring 






Seemed like he was 


Showed no emotion 






reading notes 


Seemed like he was 






Seemed like a ‘robot’ 


reading notes 






Not involved 


Seemed like a ‘robot’ 



Table 5. Overall Aspects That Made Instructor Seem Real 



Aspect of the Lesson Number of Responses 

I could see him- Video 100 

Responded to student comments and questions 86 

I could hear him 76 

Interacted with students 52 

Asked questions 33 

Changing voice pitches-tone of voice-no monotone 32 

Spoke about his family and shared personal information 3 1 

Called students by their first name 30 

Encouraged students to ask questions 26 

Used gestures-moved head-body when he was talking 27 

I could see his picture 22 

Seemed knowledgeable 2 1 

Showed personality-Expressed how he felt 20 

Used examples 18 

Caring-Seemed to care about students’ understanding 16 

Offered office hours- Asked students to come and see him- talk to him 14 

Conversational tone with students 14 

Used visuals-power point slides 1 3 

Used humor-Told jokes 8 

Made sure the students were involved 6 

Did not seem like he was reading 3 

Allowed students to call him by his first name 2 



Discussion 

Congruent with previous research examining instructor immediacy (Gorham & Zakahi, 1990) the 
findings of this study indicate that the use of instructor immediacy behaviors impacts how students perceive 
the instructor suggesting that instructors can monitor their behaviors based on the immediacy literature. The 
findings showed that video-enabled computer conferencing tools can facilitate the projection of more 
immediacy behaviors than audio-enabled computer conferencing tools; however instructors can project 
immediacy behaviors with the use of audio alone. Therefore, the findings of the present study have 
prescriptive value for training faculty to utilize the information identified in the immediacy literature not 
only for teaching face-to-face but also for teaching students at a distance. Regardless of the online 
computer conferencing environment instructors could be trained to utilize relevant immediacy behaviors 
such as encouraging students to ask questions, using humor, calling students by their first name, answering 
questions and providing feedback, sharing personal information, and so forth. 

Furthermore, the findings showed a strong, positive relationship between perceived instructor 
immediacy and perceived instructor social presence. The availability of video-enabled tools could enhance 
student perception of instructor immediacy, which according to the current study will also increase 
perception of social presence. However, in the absence of video, instructors can still project several 
immediacy behaviors and increase perception of social presence. 
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Finally, the results of the learning outcomes question is aligned with Gorham and Zakahi (1990) 
who point out that the relationship between cognitive learning and instructor immediacy is difficult to 
explain and that the relationship is non-linear. More specifically, in studies where cognitive learning was 
assessed as a test or course grade the relationship between immediacy and cognitive learning was not 
supported (Gorham & Zakahi, 1990). Many factors such as the characteristics and attitudes of the students, 
the length of the lecture, and the topic of the lecture might have affected the variance in the findings of this 
and previous studies examining learning outcomes in different learning environments. 

Overall, this study sheds light on the importance of the role of the instructor in the online 
classroom. Understanding the implications of instructor behaviors and instructional tools could elucidate 
future directions for both distance learners and on-campus learners. As different types of computer 
conferencing tools continue to evolve further research must be conducted to explore the impact of these 
tools. From a practical perspective, the choice of the communication tool may have potentially wide 
implications for leaders of educational institutions concerned with the various pedagogical and financial 
issues associated with the selection of a particular course delivery option. While social presence can 
increase student satisfaction, instructor immediacy has been shown to affect student cognitive, affective and 
behavioral learning as well as motivation. Understanding how immediacy affects social presence in online 
learning contexts may guide the design of more interactive and successful distance education courses. 
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Effects of practice in a linear and non-linear 
Web-based learning environment 

Florence Martin and Jeremy Tutty 



Instructional elements remain the foundation of current instructional design practice. Practice is the 
instructional element provided after learners have been given information required to master an objective and 
Navigation is the non-instructional element guides the learner on the sequence of instruction. Linear Navigation can 
be referred to as program control where the learners do not have control over sequence and Non-Linear Navigation 
can be referred to as Learner control over sequence of instruction. 

The purpose of this study was to investigate the effect of practice with feedback, navigation type on 
achievement, attitude, and time when students use a web-based instructional program and the interaction between 
practice and navigation type. 240 students from a large south western university participated in four different web 
based environments. Significant differences were found for practice main effect, but not for navigation main effect. 
There was no interaction between practice and navigation. 

There were significant differences for attitude items, I learned a lot from this program and the program 
gave me enough opportunity to practice between the treatments who received practice and those who did not. This 
study reinforces on the importance of practice and has implications for the design and development of web-based, 
multimedia instruction. 



Introduction 

Instructional elements 

Forty years ago, Robert Gagne published the first edition of his book The Conditions of Learning (1965) in 
which he proposed nine events of instruction that provide a sequence for organizing a lesson. These events remain 
the foundation of current instructional design practice (Reiser & Dempsey, 2002; Richey, 2000). They represent 
desirable conditions in an instructional program and increase the probability of successful learner achievement 
(Gagne, 1965, 1985, 1988; Gagne, Briggs & Wager, 1992). Other authors cite similar elements of instruction that 
promote student learning from an instructional program (Dick, Carey, & Carey, 2005; Sullivan & Higgins, 1983). 
Some of the instructional elements that have been suggested by Gagne (1985) and Dick et al. (2005) to promote 
learning are objectives, practice with feedback, examples and review. Each of these instructional elements is 
described in the following paragraphs. 

Practice and Feedback 

Practice involves eliciting performance from learners (Gagne, 1985; Gagne et al., 2005). It is often 
provided after learners have been given information required to master an objective. Practice provides an 
opportunity for learners to strengthen new knowledge by internalizing it so they can recall and use it (Foshay, Silber, 
& Stelnicki, 2003). It helps to confirm correct understanding and repeated practice increases the likelihood of 
retention (Klein, Spector, Grabowski & de la Teja, 2004; Kruse & Kevin, 1999). Practice is effective when it is 
aligned with assessment and with the skills, knowledge and attitudes reflected in the objectives (Merrill, 2002; 

Reiser & Dick, 1996). 

Feedback can be defined as “knowledge of one’s performance provided” (Delgado & Prieto, 2003, p. 73). 
Practice provides an opportunity for feedback that confirms the student’s answer as being correct or indicates that it 
is incorrect. Feedback strengthens the probability of correct responses and reduces the probability of subsequent 
incorrect responses (Philips, Hannafm & Tripp, 1988). Kulhavy & Stock (1989) define feedback as information 
consisting of two components: verification and elaboration. Verification is the simple, dichotomous judgment that 
an initial response was right or wrong. Elaboration consists of all substantive information contained in a feedback 
message. 
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Researchers have found that practice has a significant effect on performance. Hannafm (1987) reported a 
significant difference between practiced and non-practiced items on the learning of cued and uncued information 
presented via computer-based instruction. Phillips et al. (1988) found a significant difference favoring practice over 
no practice in an interactive video in which practice items were embedded questions. Hannafm, Philips, Rieber & 
Garhart (1987) noted that practice effects were more pronounced for facts than for applications in computer-based 
instruction. Participants who received intellectual skills practice in a cooperative learning environment performed 
significantly better than those who received verbal information practice (Klein & Pridemore, 1994). 

Practice provides an opportunity for feedback that confirms the student’s answer as being correct or 
indicates that it is incorrect. This feedback strengthens the probability of correct responses and reduces the 
probability of subsequent incorrect responses. (Philips et al., 1988). Simple forms of feedback are effective when 
learners are able to answer items correctly. More elaborate forms such as providing and explaining the correct 
answer and explaining why a wrong answer is incorrect are helpful when learners answer incorrectly (Kulhavy, 
1977). Simple forms of feedback are most effective for simple verbatim and verbal information types of learning 
(Kulhavy, White, Topp, Chan & Adams, 1985). 

Non-Instructional elements 

Apart from the instructional elements which Gagne (1985), Dick et al. (2005) have proposed and many 
researchers have researched on, there are the non-instructional elements such as usability, navigation type, learner 
control that also influence learning. Researchers have examined the effects of these non-instructional elements such 
as navigation type (Su & Klein, 2006), personalization (Ku & Sullivan, 2002), animated agents (Atkinson, Mayer & 
Merrill, 2005) in student achievement and attitudes. As Hannafm (1987) noted, some design strategies may have 
positive effects when used in isolation that are diminished or negated when these strategies are used in combination 
with more powerful techniques. The effects of these instructional elements could be enhanced or diminished when 
used in combination with other variables such as different navigation types, media types. 

Linear and Non-Linear Navigation 

Linear Navigation can be referred to as program control where the learners do not have control over 
sequence and Non-Linear Navigation can be referred to as Learner control over sequence of instruction. Hypertext 
has been defined as an approach to information management in which data is stored in a network of nodes connected 
by links. Shneiderman defines hypertext as "a database that has active cross-references and allows the reader to 
'’jump" to other parts of the database as desired” (Shneiderman & Kearsley, 1989). Much of the previous research on 
the effects of navigation tools look at efficiency and effectiveness of hypertext environment (Boechler & Dawson, 
2002; Dee-Lucas & Larkin, 1995; Dias & Sousa, 1997; McDonald & Stevenson, 1998). Efficiency measures are 
based on speed and the number of steps taken to complete an information search. Effectiveness measures focus on 
the user's search accuracy and the users understanding of the structure of the document (Boechler & Dawson, 2002). 

Many of the instructional programs designed to test in computer based instruction were built with 
HyperCard and Tool book initially and now it is built with Director, Authorware, Dreamweaver and Flash. In general 
these programs have been linear in format (Freitag & Sullivan, 1995; Schankenberg, Sullivan, Leader & Jones, 
1995; Martin, Klein & Sullivan, inpress). These programs do not allow learners to navigate to any screen of their 
choice except in a linear format. But with the advent of web and the hypermedia structure, programs are now built 
with the feature such that the users can trace the path they like to within these computer based programs. 

Learner control generally increases effectiveness, efficiency, and motivates learners (Reigeluth & Stein 
1983). There are no disadvantages against using learner control as long as the control option does not confuse the 
learners. There are only arguments for and against the degree of learner control. Depover & Quintin (1992) mention 
that the degree of learner control depends on variables like age, previous knowledge, learning progress, complexity 
of material and familiarity with the subject. Some of the other variables that influence the degree of learner control 
are prior knowledge, student strategy and ability, learning progress, complexity of material and familiarity with the 
subject (Depover & Quintin, 1992; Hannafm, 1984; Milheim & Martin, 1991; Steinberg, 1989). 
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Purpose of Current Study 

In the previous studies conducted by the author (author, (2007); author, inpress), the learners did not have 
control on the sequence of the instruction in the program and had only linear navigation (program control). Hence in 
order to answer the questions of the effects of the instructional element practice with feedback when learners had 
control (non-linear navigation) and learners had no control (linear navigation), this study was proposed. The IPSO 
instructional program with the same instructional content but with the changes to the navigation links was used in 
this study. 

The purpose of this study was to investigate if a) the presence or absence of practice in a web based lesson 
had significant effect on student achievement, attitude and time b) if the navigation types which provided control 
over sequence of the instructional element (Linear, Non-Linear) had a significant effect on student achievement, 
attitude and time and c) if there was any interaction between practice and navigation type. The elements investigated 
in the study, practice with feedback, and linear and non-linear navigation type were combined into four different 
versions of web based programs in a manner that permitted investigation of the effectiveness of the program when 
practice was present and absent for both linear and non-linear navigation types. 

The primary research questions for this study are listed below. 

1 . What is the effect of practice with feedback on achievement, attitude, and time when students use a web- 
based learning environment? 

2. What is the effect of navigation type (linear and non-linear navigation) on achievement, attitude, and time 
when students use a web-based learning environment? 

3. Does practice and navigation type (linear and non-linear navigation) interact to influence achievement, 
attitude and time? 

The researchers anticipated that the combination of practice and linear navigation would have higher student 
achievement while practice with non-linear navigation would have higher student attitude. 

Method 



Participants 

Participants were 240 freshman and sophomore undergraduate students enrolled in a computer literacy 
course at a large Southwestern University. 60 students participated in each treatment. The students enrolled in this 
course had varied background knowledge on computers and were from different majors including education, 
communication, journalism and others. Students participated in this study as part of the course requirement and the 
score in the post test was part of their course grade. 

To avoid the variation in treatments with in the class (practice, no practice, linear navigation, and non- 
linear navigation), the students from were assigned to the treatments by classes and not by individual. The classes 
were randomly assigned to one of the four treatments based on the pretest average scores. It was a quasi- 
experimental study due to this nature of assignment to the treatments. This was one of the limitations to the study 
but helped to avoid differences in content, attitude or time spent on the program between the students enrolled in the 
same class. 

Materials 

Four different versions of a web-based lesson on the topic Input, Processing, Storage and Output of a 
Computer (IPSO) were developed using Dreamweaver. IPSO explains the primary operations of the computer which 
are Input, Processing, Storage and Output. An introduction section was included before the primary operations were 
explained in detail. This section introduced what a computer is and classified it based on size, power and generation. 
It also explained the IPSO cycle. The next four sections described the concepts of the Input, Processing, Storage 
and Output operations in a computer and explained the function of the different components associated with that 
operation. The content used in this study was part of the required content for the course. The web-based lesson was 
pilot tested with five students before it was used in the study. 



223 




The four different versions of instructional program were as follows: 

1 . Program with Practice and using Linear Navigation (Program control) 

2. Program with Practice and using Non-Linear Navigation (Learner Control over Sequence) 

3 . Program without Practice and using Linear Navigation (Program Control) 

4. Program without Practice and using Non-Linear Navigation (Learner Control over Sequence) 

Practice Screens. 

The first two programs had practice screens in the program and it provided students with an opportunity to 
practice the content they were learning. There were a total of five practice screens, each of which contained five 
four-choice multiple-choice questions. The student received immediate feedback after each response to a practice 
item. Students had the option to practice until they got the right answer. One practice screen was presented after 
each information screen. One example from the five items on a practice screen is shown in Figure 1 and Figure 2. 



Know Your Computer mom 

Intro Input Processing Storage Output 

INPUT Objectives Intro to Input Input Devices 1 Input Devices 2 



Input Devices - Pi notice 

These practice questions are similar to the questions on the IPSO posttest. If you qet a wrong answer while practicing, 
keep tiying until you get it right. 

1. A peripheral used to transfer data from the outside world into a computer system is called 

O Input device 
O Processing device 
O Storage device 
O Output device 



Posttest 

Review 



Figure 1 : Practice screen in the non-linear navigation program 



Know Your Computer mom 

Intro Input Processing Storage Output 



Input Devices - Piactice 

These practice questions are similai to the questions on the IPSO posttest. If you get a wrong answer while practicing, 
keep tiying until you get it right. 

1 . A peripheral used to transfer data from the outside world into a computer system is called 

O Input device 
O Processing device 
O Storage device 
O Output device 



Figure 2: Practice screen in the linear navigation program 



Procedures 

Eighteen sections of students (n = 240) enrolled in the computer literacy course were blocked by classes 
and randomly assigned to the four treatment groups. The pretest, which took approximately 15 minutes to complete, 
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was administered a week prior to the study. The classes were blocked into four groups based on their mean pretest 
scores, and one class within each block was assigned to each of the four treatments. 

The participants used the web-based IPSO lesson during the sixth week of the semester. Participants met in 
a regular computer lab for instruction and were directed by the instructor to the web address for the instructional 
program. Each class was routed directly to its treatment version of the program. Students worked through the 
program at their own pace, averaging approximately one hour. Then they took the posttest and the attitude survey 
online. All six treatment groups followed the same procedure. Thus, the experimental differences in treatments 
occurred exclusively in the materials themselves and not in the procedure. 

Criterion Measures 

The criterion measures consisted of a posttest and a student attitude survey. In addition, a pretest was used 
to assess subjects’ knowledge of the content prior to the instruction. 

Pretest. The pretest consisted of 20 multiple-choice questions covering the content with four response 
choice questions. The overall mean score on the pretest was 8.38 or 42%, indicating that participants were not very 
knowledgeable about the content prior to instruction. 

Posttest. The posttest consisted of the same 20 multiple-choice questions that were on the pretest. The 
reliability of the posttest was .65. The item analysis done on the posttest revealed that question 17 was the most 
difficult with a difficulty level of .42, question 3 was at .60, followed by question 1 and 2 at .62 and .63. The rest of 
the questions difficulty level varied between .70 and .99. 

Attitude Survey. The attitude survey assessed student attitudes towards the instructional program and the 
presence or absence of the instructional events. The survey included 12 Likert- type questions that were rated 
strongly agree (scored as 4) to strongly disagree (scored as 0). The survey also included two open-ended questions 
that asked the participants what they liked best and least about the program. The survey was administered after the 
lesson and the posttest were completed. The reliability of the attitude survey was .83. 

Data Analysis 

A 2X2 analysis of variance (ANOVA) test was conducted on data obtained from the achievement posttest 
and on the total time spent on the program. The attitude survey results for the Likert type items (Items 1-6) was 
analyzed using a 2X2 ANOVA. All analyses revealed significant differences. 

Results 

Achievement 

The first research question investigated the effects of practice and navigation type on student achievement. 
Table 1 shows the mean scores and standard deviations for achievement on the pretest and posttest by treatment. The 
average pretest score was 8.46 (SD = 2.26) and posttest score was 15.91 (SD = 2.92). Participants who received 
practice and linear navigation scored the highest on the posttest (M=17.14) and those who received no practice and 
had non-linear navigation scored the lowest (M= 14.78) on the posttest. 

A 2X2 ANOVA conducted on the pretest data revealed no significant difference for practice main effect, 
navigation main effect or interaction. 2X2 ANOVA conducted between the treatment groups on the posttest 
revealed a significant practice main effect, F (1, 196) = 22.388, p < 0.01. Thus, there was a significant difference 
between the groups that received practice and no practice. Those who had received practice (M= 16.84) performed 
significantly higher on the posttest compared to those who did not receive practice (M=14.98). However, there was 
no significant difference for those who had control over the instruction using the Linear navigation (M= 15.66) and 
Non-Linear navigation (M=16.16). There was also no significant interaction between practice and navigation type. 
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Table 1: Means and Standard Deviations (SD) for Posttest scores by treatment 







Pretest 






Posttest 




Treatment 


Linear 

Navigation 

Mean 

(SD) 


Non-Linear 

Navigation 

Mean 

(SD) 


Total 


Linear 

Navigation 

Mean 

(SD) 


Non-Linear 

Navigation 

Mean 

(SD) 


Total 


Program 


8.24 


8.50 


8.37 


17.14 


16.54 


16.84 


with 

Practice 


(2.75) 


(2.32) 


(2.53) 


(1.92) 


(3.42) 


(2.77) 


Program 


8.62 


8.48 


8.55 


15.18 


14.78 


14.98 


without 

Practice 


(2.60) 


(2.21) 


(2.40) 


(2.81) 


(2.77) 


(2.78) 


Total 


8.43 


8.49 


8.46 


16.16 


15.66 


15.91 




(2.66) 


(2.25) 


(2.26) 


(2.59) 


(3.22) 


(2.92) 



Attitude 



The next research question dealt with the effects of practice and navigation type on student attitudes. Table 
2 shows means for responses to the 6 Likert-type items on the attitude survey. The items were rated on a 5 point 
Likert scale from strongly agree (N= 4) to strongly disagree (N=0). 



Table 2 Means for Attitude survey by treatment 



Treatment 


Program with Practice 


Program without Practice 




Linear 


Non-Linear 


Total 


Linear 


Non-Linear 


Total 




Navigation 


Navigation 




Navigation 


Navigation 




Attitudes Items 


Mean 


Mean 




Mean 


Mean 


Mean 




(SD) 


(SD) 




(SD) 


(SD) 


(SD) 


1 . 1 learned a lot from this program. 


3.32 


3.22 


3.27 


3. .06 


3.12 


3.09 




(.587) 


(.648) 


(.617) 


(.620) 


(.659) 


(.637) 


2.The overall quality of the program was 


3.36 


3.26 


3.31 


3.26 


3.22 


3.24 


good 


(.525) 


(.600) 


(.563) 


(.600) 


(.507) 


(.553) 


3. 1 would recommend this program to 


3.22 


3.10 


3.16 


3.04 


2.94 


2.99 


other students 


(.648) 


(.678) 


(.662) 


(.676) 


(.586) 


(.631) 


4. 1 would enjoy using other computer 


3.00 


3.04 


3.02 


2.96 


2.94 


2.95 


programs like this one in future lessons 


(.736) 


(.755) 


(.742) 


(.735) 


(.712) 


(.720) 


5. The program gave me enough 


3.20 


3.26 


3.23 


2.56 


2.40 


2.48 


opportunity to practice what I was learning 


(.639) 


(.694) 


(.664) 


(.760) 


(.782) 


(.772) 


6. The program gave me enough control to 


3.22 


3.36 


3.29 


3.12 


3.30 


3.21 


move around the program. 


(.616) 


(.663) 


(.640) 


(.718) 


(.544) 


(.640) 


Total 


3.22 


3.21 


3.21 


2.99 


2.99 


2.99 



A MANOVA conducted on the overall attitude data revealed a significant difference on the 6 attitude 
questions, F (18, 535.06) = 4.33, p < 0.01. Follow-up 2x2 ANOVA conducted on the attitude data indicated 
significant differences for items 1(1 learned a lot from this program) and 5 (The program gave me enough 
opportunity to practice) between the treatments who received practice and those who did not. No items showed 
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significant difference for the navigation type (linear versus non-linear navigation). There was no interaction between 
practice and navigation type on the attitude data. For both the above items that had practice main effect, the 
participants who received practice had rated it significantly higher than those who did not receive practice. 

The attitude data also showed that participants who used the program with practice had higher attitudes and 
had rated higher on all the six items compared to those who did not receive practice. Item 2 “The overall quality of 
the program was good” was the rated the highest (M = 3.28) by both the practice and no practice group. Those who 
did not receive practice recognized the absence of practice in their programs and rated item no 5 “The program gave 
me enough opportunity to practice what I was learning”, the lowest (M = 2.48). There weren’t much difference in 
attitudes of the participants when comparing the navigation method they received and were almost equivalent in 
their ratings based on navigation. 

The attitude survey also included two open-ended questions that asked the participants what they liked the 
best and least about the program. The four most frequent responses for what participants liked the best in the 
program were (1) the practice questions (n=65) (2) clear navigation and structure (n=45) (3) the review section 
(n=43) (4) graphics, animations and visuals (n=34) (4) Highly informative (n=30). The most frequent response for 
what parts liked the least were (1) Very long program (n= 36) (2) Lot of information (n=23). 

Time 

Table 3: Means and Standard Deviations (SD) for Time spent in minutes by treatment 





Time 


Treatment 


Linear 


Non-Linear 


Total 




Navigation 


Navigation 






Mean 


Mean 






(SD) 


(SD) 




Program with 


35.98 


35.14 


35.56 


Practice 


(13.59) 


(18.84) 


(16.34) 


Program 


31.00 


31.66 


31.33 


without 

Practice 


(13.15) 


(16.19) 


(14.68) 


Total 


33.49 


33.40 


33.44 




(13.53) 


(17.56) 


(15.64) 



On calculating the time spent in the program, there were no significant differences between the groups 
based on practice and navigation. Those who received practice spent more time on the program (M = 35.56) than 
those who did not receive practice (M = 3 1.33), but there were no significant differences. Both the navigation types, 
linear (M=33.49) and non-linear (M=33.40) spent about the same amount of time in the program. 

Discussion 

This study examined the effects of Practice and Navigation type (Linear and Non-Linear) on achievement, 
attitude and time. College students enrolled in a computer literacy course used a web-based lesson delivered on the 
web to learn about input, processing, storage and output of a computer (IPSO). The computer based lesson included 
multiple choice practice items and two types of navigation (Linear, Non-Linear). Linear navigation directed them 
from one page to the next where as in Non-Linear navigation they had the freedom to navigate in any path. They had 
control over the sequence of instruction. Results indicated that there was a significant difference between the groups 
that received practice and no practice, but there was no significant difference in Linear and Non-Linear navigation. 
There was no significant interaction between practice and navigation type. 
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Achievement 



Practice resulted with a significant difference both in achievement and in attitudes. Practice provides an 
opportunity for learners to confirm their correct understanding, and repetition increases the likelihood of retention of 
new knowledge (Kruse & Kevin, 1999). In this web-based lesson, participants who received practice were also 
given feedback which confirmed the student’s answer as being correct or indicated that it was incorrect. This 
feedback strengthens the probability of correct responses and reduces the probability of subsequent incorrect 
responses. (Philips, Hannafm & Tripp, 1988). The confirmation of the answer during practice increased the 
likelihood of retention of the content. Presence of practice results in interaction between the lesson and the learner. It 
is effective in performance when it is aligned with the assessment in the form of a posttest and with the skills, 
knowledge and attitudes reflected in the objectives (Reiser & Dick, 1996). In this web-based lesson, practice was 
directly aligned with the posttest and with the information presented. The findings of the current study is consistent 
with previous research on computer based instruction that found practice had an effect on learning (Hannafm, 1987; 
Hannafm & Tripp, 1988). 

Navigation types (linear, non-linear) did not result in a significant difference. This could have been due to 
the fact that even though the non-linear treatment had the flexibility to take any path that they decided, the computer 
based lesson was well structured and organized. The lesson was designed using all the instructional elements and 
was instructionally aligned from objectives to assessment. The results could have been different, if the instructional 
material was not well structured and did not have the other instructional elements or instructional alignment. Though 
there was no significant difference, participation of the linear navigation program scored higher than the participants 
of the non-linear program. 

Participants who received practice and linear navigation scored the highest on the posttest and those who 
received no practice and had non-linear navigation scored the lowest on the posttest. When enough of structure is 
provided and the instructional material is well designed, the students do better when the navigation is linear and they 
are forced through every screen where they learn from every instructional element such as objective, practice, 
feedback and review. The absence of the instructional element practice and non-linear navigation resulted in the 
lowest posttest scores. 

Attitude 



The attitude survey had 6 Likert-type items. Participants who received practice in their computer lessons 
had higher attitudes compared to those who did not receive practice. Those who received practice had an 
opportunity to interact with the web-based lesson and it helped them perform better and have higher attitudes. 
Higher attitudes of the participants who received practice could have been due to the feedback they received during 
practice, which strengthened the probability of correct responses and reduced the probability of subsequent incorrect 
responses. The attitude results remained consistent with the achievement results. 

Item 1 “I learned a lot from this program” and Item 5 “The program gave me enough opportunity to 
practice what I was learning” resulted with significant differences between the treatments which received practice 
and which did not receive practice. The significant difference on item 5 (The program gave me enough opportunity 
to practice what I was learning) reveals that the participants realized the presence of practice in their computer based 
lessons. And also on item 1 (I learned a lot from this program ), those who received practice had higher attitudes that 
they had learned a lot from the program. The presence of interactive well aligned practice items, which provided 
feedback and corrected their understanding of the concepts, must have been the reason for them to state that they 
had learned a lot from this program. Item 6 “The program gave me enough control to move around the program” 
which is about the navigation aspect of the program did not result in significant differences. Thus both in attitudes 
and achievement, there was significant differences for the presence of practice but not for navigation type. 

In the open ended section, practice topped the list for what the participants liked the best about the program 
and was followed by the clear structure and navigation. It can be noted that participants realized that practice made a 
difference in the program. We may not have had a significant difference in the navigation types, but from the open 
ended question it is shown that students were aware of the structure and navigation used in the program and had 
rated it as the second best feature in the program. 
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Time 



Time did not result in any significant difference for both practice and navigation type. Though the 
participants in the practice treatment spent longer time in the program, it was not significantly different from the 
treatment who did not receive practice. But the time spent by both the navigation types were almost the same. 
Participants in the linear navigation treatment spent (M=33.49) minutes and those in the non-linear navigation 
treatment spent (M=33.40) minutes. This shows that the different navigation types did not matter in regard to time 
spent. 

Conclusion 

This study has once again reinforced the importance of practice in web based/computer based lessons. It 
once again confirmed Practice to be effective in help in student achievement and attitude. If the lesson is 
instructionally sound with important instructional elements such as practice then irrespective of the non-instructional 
element such as navigation type, learners have high achievement and attitudes. It also showed that when the lesson 
is well structured then the effect of navigation is not seen. 

This study has implications to the design and development of web-based instructional material. Practice is 
an effective instructional element for enhancing student achievement. This suggests that it should be included in 
web-based instruction especially when students are tested using items aligned with the objectives and practice items. 
We also recommend including different types of practice items. In this study, multiple choice practice items with 
immediate feedback to students were included. Also, the content was more on learning facts and concepts. Future 
research should focus on variation in the instructional content and the type of practice and feedback involved. 
Additional research should examine how instructional elements in computer-based instruction influence outcomes 
such as problem solving and complex learning tasks. As was done in this study, research in these settings should 
include measures of student achievement, attitudes and time. Further studies can be conducted to test the 
effectiveness of the other instructional elements such as objectives, examples, and review along with the different 
non-instructional elements such as intelligent agents and other usability elements including navigation types. It will 
be helpful to measure the practice scores and their correlation to the posttest scores. Studies of this nature will 
continue to inform designers about the influence of instructional elements on learning and performance. 
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Open and Distance Learning Opportunities 
and e-Learning Services in Turkey 

M. Emin Mutlu 



Abstract 

This study analyzes the open and distance learning opportunities at various levels starting from primary 
education to bachelor’s education offered to individuals having needs for education and tries to determine the 
potential for using e-Learning in the Turkish open education system as a whole evaluating the e-leaming 
experiences at Anadolu University Open Education System between 1999-2007. 

Key Words: open and distance education, e-leaming 

Introduction 

Open education is one of the ways of offering equal opportunity in education to people who cannot find 
the opportunity to get face-to-face education for different reasons. People may prefer open education because 
they are working, they have physical disabilities, they are imprisoned, they cannot leave their home because 
there maybe people they are responsible for caring, they do not meet the age criteria put by traditional 
universities, they are doing their military service, or because of low income. In addition, housewives, the ones 
who would like to get a second degree, and the ones who live abroad prefer open education. 

Since students at open education system may not participate in traditional face-to-face education, the 
most frequently used teaching method that is used in the open education is ‘distance education’. The distance 
education was based on teaching via letter and newspaper many centuries ago, and technologies such as 
telephone, radio, and cinema were also used in this century. Education via television made Open Education 
famous today, and new distance education establishments were created in 1980’s in many countries using the 
experiences of the Open Education of England, which was opened in 1960’s. 

Turkish National Education System 

The Turkish National Education System is composed of four steps as the pre-school teaching for 0-6 
age group, primary education for 6-14 age level, high school for 14-18 age group and higher education for the 
age of 18 and above. The primary education, which lasts five years, was obligatory to 1997, and since 1997 the 
length of the obligatory education has become eight years for 6-14 age level. The length of high school was 
increased three years to four years in 2006 (OSYM, 2007) (Fig. 1). 

Institutions Offering Open and Distance Education in Turkey 

The open education system in Turkey has been started by the Anadolu University as a central 
application in Higher Education in 1982. In the first year, 29.000 students were enrolled in two programs, and 
today, students are accepted to 15 associate degree and 7 graduate level programs on the basis of students’ 
grades in the open education system. 1.100.000 students are enrolled in the Anadolu University Open Education 
system, and more than 900.000 of those students are active. More than 45% of those students study in the 
Anadolu University. 

The right to offer open education in Turkey at associate and bachelor’s level in from a specific center has 
been given to Anadolu University. Anadolu University offers this service via three faculties and a center (AU, 
2007). 



• Open Education Faculty (Open Associate and Open Bachelor’s Degree) 

• Faculty of Business Administration (Open bachelor’s education) 

• Faculty of Economics (Open Bachelor’s Education) 

• Applied Research Center for Continuous Education (Open e-Certificate Education) 

The Open Education High School system was started in 1992 by the Ministry of National Education (MNE) 
for students who are above the secondary education age. Today, the open high school has more than 300.000 
students and more than 275.000 of them are active. The Open Primary School system was started by the Ministry 
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of National Education in 1997 for the ones who could not complete their primary education and adults who are 
above the compulsory primary education age level so that they can receive education everywhere and in all 
circumstances through distance education principles and techniques. The Open Primary School has more than 
250.000 students, of which more than 100.000 are active. The basic instructional materials in the Open Primary 
School and Open High School that depend on the Ministry of National Education are the course books and 
television/radio programs. The Vocational Open Education High School that use distance education and face-to- 
face education methods has been started by the MNE in 2006 and today more than 78.000 students are enrolled 
in Industrial Vocational High Schools, Imam and Preachers’ High School, Vocational Girl's High School and 
Trade Vocational High School. The MNE Vocational and Technical Open Education School was established in 
1974 and offers distance education for certificate. Nearly 70.000 individuals graduated from the Vocational and 
Technical Open School have been given a document stating that they are authorized to work as electricians 
(MNE, 2005, 2007). 
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Figure 1. Turkish National Education System 



The open education institutions within the MNE are conducted by the General Directorate of 
Educational Technologies (EGITEK, 2007) (Fig. 2). 

The aim of the Turkish National Open Education System is to offer education from primary education to 
bachelor’s education mostly to individuals who could not be involved in the on-campus education system or who 
were excluded from the system. 
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Figure 2. Turkish National Open Education System 

Other than the Open Education System, 8 universities in Turkey have Internet-based associate degree, 
bachelor’s degree, and master’s degree programs. Those schools are inspected by the Distance Education 
Commission within the Higher Education Council. Since the acceptance terms for those programs are the same 
as the acceptance terms for face-to-face programs and the education is offered according to the laws for face-to- 
face education, they cannot offer a real open education. Therefore, they are not included in the scope of this 
study. 



• Anadolu University (1 Associate, 3 Master’s programs) 

• Middle East Technical University (1 Master’s programs) 

• Sakarya University (5 Associate, 1 Master’s programs) 

• Mersin University (4 Associate programs) 

• £ukurova University (1 Associate program) 

• Gazi University (2 Associate programs) 

• Maltepe University (3 Associate, 1 Master’s programs) 

• Bilgi University (1 Master’s program) 



Options for Entering to the Open and Distance Education 

In Turkey, individuals are offered open education at primary, secondary and higher education levels. 
Acceptance to the Open Primary and Secondary Education are supervised by the MNE and acceptance to the 
Open Higher Education is conducted by the Student Selection and Placement System (SSPS) organized by the 
Student Selection and Placement Center (OSYM, 2007). Two types of placements are available for the SSPS 
open education programs: a) Graduates of the vocational and technical high schools can be placed to the open 
associate degree programs related to their field without taking any exams, b) Individuals receiving 160 score and 
more in the Student Selection Exam can be placed into any open associate and bachelor’s program without any 
quota. The ones who cannot be placed into any program by the Student Selection and Placement System can be 
enrolled in Open Education e-Certificate programs and receive education to get a certificate. 
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There are five ways of being accepted to the open education in Turkish education system. 

1. Individuals who could not complete their face-to-face primary education can go to the open primary 
school. 

2. Individuals who completed their primary education can go to the open education high school or 
vocational high school. 

3. High school graduates and individuals who could not be placed into a higher education program by the 
Student Selection and Placement System can be enrolled in open e-certificate programs. 

4. High school graduates and the ones who obtain 160 or higher score in the Student Selection Exam can 
be placed to an open associate degree or bachelor’s degree program. The ones placed to the on-campus 
associate degree or bachelor’s program can transfer to the open education system through “vertical 
transfer”, “bachelor’s degree completion”, or “second university” opportunities. 

5. Graduates of vocational and technical high schools can be placed to the open associate degree programs 
by the SSPS. The ones placed to the on-campus associate degree programs without taking an exam and 
having sufficient requirements can transfer to the open education system through “vertical transfer”, 
“bachelor’s degree completion 44 and “second university” opportunities. 

The road maps offered to individuals to enter to the open education system are explained in the following 
sections. 

Open and Distance Education Opportunities for Individuals Over 15 Year of Age 

Individuals who could not continue their education after the first five years or who could not complete 
their face-to-face education can register in the 6 th , 7 th or 8 th classes of the Open Primary School. The ones who 
complete the Open Primary School receive a primary school diploma. They obtain the right to go to the Open 
Education High School and Vocational Open Education High School (Fig. 3). 




Figure 3. The open and distance learning opportunities for individuals over 15 year of age. 

Individuals having the qualifications below can apply for the Open Primary School (OPS, 2007). 

• Individuals who started the age of 15; 

• 5 -Year programmed primary school graduates, 

• Individuals who finished the 5 th class of the primary schools and who left from the 6 th , 7 th or 8 th 
class, 

• Adults who completed reading-writing courses of the public education centers, 

• The ones studied abroad and having equivalence for the 6 th , 7 th or 8 th grade. 

• If they meet one of the conditions mentioned above, the ones who prove that they are imprisoned or 
physically handicapped and the ones who prove that they need special education through a 
document that can be received from the Special Education Services Committee can apply. 

Open and Distance Learning Opportunities for Individuals Who Completed Their Primary Education 

Individuals who cannot receive face-to-face education in secondary education institutions can register in 
the Open Education High School, Vocational Open Education High School or Vocational and Technical Open 
Education School. The ones graduated from the Open Education High School are considered as the graduates of 
a general high school, and graduates of the Vocational Open High School are considered as if they graduated 
from a vocational high school (Fig. 4). 
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Figure 4. Open and Distance Learning Opportunities for Individuals 
Who Completed Their Primary Education 

Individuals having the following qualifications may apply for the Open Education High School (Grades 
9, 10, 11, 12) (OEHS, 2007) 

• Graduates of primary school/secondary school, 

• The ones who left from the general high school programs within MNE or other ministries, 

• Individuals who left from Anatolian Fine Arts High School, Sports High School, and Social 
Sciences High School, 

• Individuals studied abroad and having the requirements mentioned above. 

Individuals having the following qualifications can apply for a Vocational Open Education High School 
(grades 9, 10, 1 1, 12.) (VOEHS, 2007). 

• Primary/Secondary school graduates, 

• Individuals who left from secondary schools within MNE or other ministries or graduates of those 
schools. 

• Individuals who studied abroad if their education level stated on the equivalence document meets 
one of the conditions mentioned above. 

Vocational and Technical Open Education School (VTOES, 2007) 

• It is an institution that teaches professions to individuals graduated from a primary school or higher 
level schools and that uses distance education or face-to-face education when necessary. 

Open and Distance Education Opportunities Offered to High School Graduates and Who could not be 
placed to Higher Education by SSPS 

Graduates of general high schools or vocational and technical high schools can be accepted to open e- 
Certificate programs organized three times a year and taught using distance education methods if they cannot be 
placed to university as a result of the centrally conducted student selection and placement exam (ECP, 2007) 
(Fig. 5). 
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Figure 5. Open and Distance Education Opportunities Offered to High School Graduates 
and who could not be placed to Higher Education by SSPS 

Anybody possessing a secondary school or higher level diploma can apply for the open education e- 
certificate programs. Twenty-two certificate programs are available for the time being. 

Open and Distance Learning Opportunities for High School Graduates and Individuals Placed to a 
Higher Education Program by SSPS 

Graduates of open general or vocational and technical secondary school and who receive at least 160 
score and higher in the SSE (student selection examination) conducted by the SSPS can be placed to open 
associate and bachelor’s degree programs. The ones who have been placed to face-to-face associate degree or 
bachelor’s degree programs can transfer to open education thanks to ‘vertical transfer in open education’, 
‘bachelor’s degree completion in open education’ and ‘second university in open education’ opportunities. 

Those individuals are offered different opportunities depending on whether they are placed to open or 
face-to-face programs or open vocational or bachelor’s degree programs by the SSPS. Those opportunities are 
examined in the following sections (Fig. 6). 
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Figure 6. Open and Distance Learning Opportunities for High School Graduates and 
Individuals Placed to a Higher Education Program by SSPS 



a) Placement to the Open Associate Programs by the SSPS 

In the 2007-2008 academic year, students are accepted to 15 open associate degree programs via the SSPS. 

Individuals placed to those programs receive an associate degree diploma when they complete those 
programs. Those graduates can register in an open bachelor’s degree program suitable to them trough vertical 
transfer opportunity and either directly continue the program or after studying one year in the preparatory class. 

Vertical transfer to the Open Education System is realized without taking an exam. Open education students 
can continue to the on-campus bachelor’s degree programs if they obtain a required score in the vertical transfer 
exam. In this study, the priority is given to the open education opportunities offered to individuals and hence not 
all the on-campus education opportunities are examined. 

b) Placement to the Open Bachelor’s Degree Programs by the SSPS 

In the 2007-2008 academic year, students will be accepted to 7 open bachelor’s program. 

Individuals who completed the first two years of the open education bachelor’s degree programs and left the 
program can find the ‘bachelor’s degree completion’ opportunity. This bachelor’s degree completion is available 
for many associate degree programs. Individuals completing the first two years of the universities offering face- 
to-face education and leaving the program after receiving an associate degree diploma can complete their 
‘bachelor’s degree’ in the open education system. 

Individuals can obtain a bachelor’s degree via either vertical transfer or bachelor’s degree completion. 
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c) Placement to the On-Campus Associate Degree Programs by the SSPS 

Individuals placed to an on-campus associate degree program can register in the open bachelor’s degree 
programs thanks to the ‘vertical transfer’ opportunity after completing the program and receive an associate 
degree diploma. While studying in an associate degree program or after their graduation from such a program, 
individuals can enroll in open associate degree programs and can enroll in an open bachelor’s degree program 
via vertical transfer after completing the program. 

d) Placement to the On-Campus Bachelor’s Degree Programs by the SSPS 

Individuals who are placed to an on-campus bachelor’s degree program can register in open bachelor’s 
degree programs after receiving their associate degree diploma after completing the first two years thanks to 
“bachelor’s degree completion in open education” opportunity. While they are studying in a face-to-face 
bachelor’s degree program or after graduation, individuals can register in open associate or bachelor’s degree 
programs thanks to “vertical transfer in open education” or “bachelor’s degree completion in open education” 
opportunities. 

e) Case Of Not Being Placed To On-Campus Education Institutions By The SSPS 

Candidates who could not be placed to face-to-face programs by the SSPS can be accepted to a program 
they like if there is free place offered by SSPS. Additional placement can be done to the open education 
programs without quota restriction. People can register in an open associate degree or bachelor’s degree program 
using this opportunity 



Open and Distance Learning Opportunities Offered To Vocational and Technical School Graduates 

Graduates of vocational and technical schools are placed to an associate degree program in their field in 
the Anadolu University Open Education System without taking an examination, which is an application 
organized by the SSPS (Fig. 7). 
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Figure 7. Open and Distance Fearning Opportunities Offered To Vocational 
and Technical School Graduates 

Individuals who completed the open vocational school can transfer to an open bachelor’s degree 
program using the ‘vertical transfer in open education’ opportunity. 

Thanks to all these opportunities, people who could not start or complete their face-to-face education 
can get the opportunity to complete their education. Individuals who complete their education thanks to open 
education can continue their face-to-face education at a higher level if their success level is enough. The ones 
who cannot manage to pass to face-to-face education can continue to study through open education until they 
receive a bachelor’s diploma. Individuals receiving a diploma from an open bachelor’s program have the right to 
enter a face-to-face master’s or doctoral program if their success level is sufficient. Opportunities such as 
“additional placement to open education programs by SSPS”, “Transfer to open education programs without 
examination by SSPS”, “vertical transfer in open education”, “bachelor’s degree completion in open education” 
and “bachelor’s degree completion in open education” and “second university in open education” adds flexibility 
to Turkish Open Education System and enhances the transfer opportunities between face-to-face education and 
open education. 
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e-Learning in Open Higher Education 



The initial design of the open higher education system of the Anadolu University includes five 
components i.e., course books designed according to distance education techniques, instructional programs 
broadcasted in the national television channel, face-to-face academic facilitation courses given at evenings and 
week-ends, central examinations organized simultaneously in all cities, and bureau services offered to students. 
Since 1999 a series of e-leaming services has been designed and offered to students in the Internet environment. 
The first service in the Internet environment was the trial exams composed of randomly chosen questions. This 
service later was entitled as e-Exam. Then e-Practice materials, interactive multimedia software, including text, 
picture, animation, and voice, have begun to be offered. The following service was e-Book that was actually 
electronic copies of the course books published on the Internet. The e-Television service, i.e., publishing the 
digital video files of the instructional programs that are published on the Internet, has begun as a result of the 
developments in the Internet infrastructure in the country and increase in the number of students having 
broadband Internet access. In the following period, e-Consulting service that allowed students ask questions to 
the subject area specialists and receive reply asynchronously was offered. In the same period, the e- Audio Book 
service, which means publishing radio-phonically vocalized digital audio files of the course books on the 
Internet, has begun. For all those services, the e-Support application that allows students to get help from experts 
about technical problems via the Internet has been put into practice. Those services that were developed by 
different teams at different times were combined under the name of Open Education e-Leaming Portal in 2005. 
This allowed students to access all services once they log in. In 2006, a multimedia e-Course design that 
included text, picture, animation, voice, and video has developed and courses have begun to be developed 
according to this design for large number of students. Video presentations based on the Flash Video technology 
are used in this design that requires a larger Internet band width than other services (Mutlu and Ozkul, 2006). 

Various e-leaming environments have been designed for each course in the Anadolu University, Open 
Education e-learning system and hence students having different learning styles are offered different options so 
that they can use the most suitable learning environments for their needs. Such an approach allows creating 
teams having different types of production processes and allows those teams to design and develop various 
learning environments that are parallel to each other. 

Although it was not compulsory students use the Anadolu University e-Leaming Portal frequently. In 
the e-leaming portal where more than 2 million students including graduates have the right to log in using their 
TR identity numbers, more than 635.000 different students logged in more than 19, 5 million times between May 
2005- September 2007. During this time period, 1.146 million hits and 163 TB data transfer occurred. The 
average visit time was observed as 26 minutes. Research shows that students using e-learning services are more 
successful than the ones who do not use it. 

Anadolu University e-Leaming Portal is the largest portal of Turkey in terms published content and the 
copy right of the content belongs to the university, in terms of the number of users accessing this content and in 
terms of the number of access. Anadolu University owns the server and band width infrastructure that is able to 
offer those services to 50.000 students at any time and 500.000 different students in a day. This capacity is 
expected to double in one year. 

e-Learning in Open Primary School and Open High School 

e-Leaming services belonging to Open Primary School, Open High School, Vocational Open High 
School and Vocational Technical Open School is offered by EGITEK unit which has been found for this purpose 
by the ministry. EGITEK prepares the following e-leaming materials for students enrolled in Open education 
institutions within the ministry (EGITEK, 2007). 

• PDF files of the courses are accessible on the Internet. 

• Television-radio programs are sold in cassette and CD format. 

• Television programs are published on the Internet via the Internet-TV service. 

• Radio programs are broadcast on the Internet via the Internet-Radio service. 

Anadolu University and Ministry of National Education should coordinate so that e-Leaming 
experiences of Anadolu University are evaluated by the MNE and for the production of e-Courses for MNE open 
education institutions and for the design of projects to deliver those courses via a learning management system. 
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Conclusion 



In the open education system of Anadolu University 1.100.000 students are offered e-Exam, e-Book, e- 
Television, e-Practice, e-Facilitation, e-Support, and e-Voiced Book services while 628.000 students in the MNE 
open education schools are offered only e-Book and e-Television services. EGITEK should take the initiative to 
diversify and increase the e-leaming services offered in MNE open education institutions. This will help offer an 
effective continuous open education to individuals from primary school to university. Investigating the national 
open education systems of other countries will allow to prepare the inventory of open education opportunities 
offered to individuals. 
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Abstract 

The purposes of this study are to create a Structural Equation Model for explaining the factors which 
influence the effects of e-leaming perceived by elementary school students, to examine the model fit of the 
Structural Equation Model, and to explain the structural relationship among the factors. 

To this end, the theoretical framework was developed to explain elementary students’ perceived effects of 
e-learning, based on critical analysis of the previous studies. Second, the model fit of the measurement model based 
on the theoretical framework was tested, using the confirmatory factor analysis and structural equation 
modeling(SEM) analysis. Third, the hypothesized structural equation model was revised and refined as the final 
model for the most appropriate one. Finally, the paths and effect size of the factors which influence the students’ 
perceived effects of e-leaming were analyzed. 

The result of the confirmatory factor analysis for measurement model pointed out that the fit of the AGFI 
did not satisfy the statistical criterion. Accordingly, Squared Multiple Correlations(SMC) was conducted, and 6 
factors were eliminated to revise the measurement model. The second test for the revised measurement model was 
conducted. The result indicated that the fit of indices in the SEM were higher than the previous model with 
X2=2, 401.83, df=979, GFI=0.913, AGFI=0.900, TLI=0.938, CFI=0.943, and RMSEA=0.036. Both credibility and 
validity were examined and turned out to be appropriate. 

Next, the SEM analysis was also conducted to verify the appropriateness of the hypothesized stmctural 
equation model based on the revised measurement model. The result indicated that the entire model was appropriate, 
but GFI and AGFI still did not satisfy the desired statistical criterion. The result led to finding the more appropriate 
model than the present model. The hypothesized stmctural equation model, therefore, was revised according to both 
the previous studies and the modification indices, using AMOS 5.0. The revised model was eventually decided as 
final, indicating a more appropriate fit than the previous model. The model fit of the final stmctural equation model 
presented x2=2, 776.39 (p=0.000), df=999, x2/df=2.779, GFI=0.900, AGFI=0.887, TLI=0.924, CFI=0.929, and 
RMSEA=0.039. 

In conclusion, this study presented that the theoretical framework and the final stmctural equation model 
were found to be appropriate for explaining the factors which influence elementary students’ perceived effects of e- 
learning. This study also added new findings to the relevant studies in that the characteristics of e-leaming 
environments caused the effects of e-leaming in the characteristics of the learners as a parameter. 

Finally, the findings about the paths and effect size of the factors which influence the effects of e-leaming 
in this study can be used to design e-leaming service which is most appropriate to the students’ abilities and to 
understand the processes of the elementary students’ learning and perceived effects during e-learning. 

Introduction 

The term “e-learning” started to be widespread in Korean elementary and secondary education after it was 
used in the Education Ministry’s announced measures in 2004 to reduce private tutoring expenses by normalizing 
public education (The Ministry of Education and Human Resources, 2004). E-leaming, therefore, has a relatively 
short history in the Korean education field. 

Now there is a move to broaden the concept of e-leaming in Korea as the government has been actively 
working to use e-leaming in teaching and learning in elementary and secondary schools (Jang Sang-hyeon et al. 
2004; Han Geon-woo, Song Gi-sang, Lee Yeoung-jun, 2005). This has caused varied terms such as cyber learning, 
online learning, and internet learning, commonly used in elementary and secondary schools, to be all replaced by 
the term e-leaming (Byeon Yeoung-gye, Kim Gyeong-hyeon, 2003). 

The trend has made e-leaming a familiar teaching and learning method for the elementary and secondary 
education. As part of its policies, the Education Ministry is providing e-learning services nationwide including “EBS 
Lectures for the Scholastic Ability Test” and “Cyber Home Learning” for elementary and secondary school students. 
These services are aimed at expanding education beyond the classroom into cyberspace; enabling students to learn 
according to their own levels and have greater right to learning choices; and enhancing the quality of public 
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education by providing students with self-directed learning opportunities that enable them to study whenever 
necessary. 

Despite the enthusiasm surrounding e-leaming, few efforts are being made to prove the effectiveness of e- 
leaming in elementary and secondary education. Some are even voicing concern about its effectiveness (Han Geon- 
woo, Song Gi-sang, Lee Yeoung-jun, 2005). 

Park Innwoo (2004) said that e-learning, by nature, is self-directed learning. This means that a more careful 
and effective approach is necessary when providing e-leaming service to elementary and secondary schools than to 
universities and companies as schoolchildren are less capable of directing their own learning than adults (Song 
Sang-ho et al., 2005). Han Geon-woo, Song Gi-sang, and Lee Yeoung-jun (2005) stressed the need to make constant 
efforts to enhance the effectiveness of e-learning in their research on “Ways to improve EBS lectures for the 
scholastic ability test for the good of the public education,” where they argued that the EBS lectures failed to take 
advantage of e-learning by conducting one-way lectures, and pointed out the lack of lecture information and 
evaluation. These points indicate the urgency of taking a systematic and comprehensive look at the effects of e- 
leaming used in elementary and secondary education. 

Theoretical Background 

Every study has different yardsticks for measuring the effectiveness of e-leaming. Many studies focusing 
on its academic effects adopted learners’ academic achievement as the major yardstick (Kang Ok-mi, 2001; Kim Se- 
eun, 2002; Lee In-sook, 2002; Moore & Kearsley, 2005; Neuhauser, 2002; Hiltz, 1990), while others used learners’ 
satisfaction (Park Seong-ik, Yoon Soon-gyeong, 2000; Thurnond et al., 2002). Some studies took learners’ academic 
achievement, participation, and satisfaction into account to explain the effectiveness of e-leaming (Kang Yeoung- 
hwan, 2004; Seo Hye-jeon, 2001; Lim Jeong-hoon, Lee Hang-nyeong, 2003; Yoo Pyeong-jun, 2003b). 

This study defines the effectiveness of e-learning as individual learner’s comprehensive and subjective 
perception of the educational experience and benefits gained through e-leaming, including their academic 
achievement, participation and satisfaction. Based on this definition, it examines existing studies and relevant 
documents for the theoretical background to find that most studies on the effectiveness of e-learning dealt with 
individual learner’s characteristics and e-leaming environmental characteristics as factors influencing effectiveness. 
The findings will be explained in detail below. 

Kang Myeong-hee (2004) argued that computer-based learning is not suitable to all learners, and that 
benefits and level of satisfaction can vary according to learners’ characteristics. This means that learners’ 
characteristics determine whether e-leaming is effective. Relevant studies focused the individual characteristics that 
affect the effectiveness of e-leaming. The characteristics identified include learning motivation, achievement 
motivation, academic self-efficacy, computer self-efficacy, learners’ familiarity with technology, learners’ desire 
and ability to direct their own study, learners’ educational background or preliminary education, learners’ 
personality, learner’s interest, time management strategies, positive attitude, test anxiety, cognitive strategies, and 
meta-cognitive strategies (Kang Ok-mi, 2001; Kim Se-eun, 2002; Song Sang-ho, 2000; Yang Yeon-sook, Yoo 
Pyeong-jun, 2003; Eom Woo-yong, Choi Eun-hee, 2001; Lee In-wook, 2002, 2003; Lee Ji-yeon, 2004; Lim Jeong- 
hoon, Lee Hyang-nyeong, 2003; Yoo Pyeong-jun, 2003a; Cho Il-hyeon, Lim Gyu-yeon, 2002; Ju Yeoung-ju, Moon 
Ja-yeoung, 2004; Compeau & Higgins, 1995; Hasan, 2003; Houle, 1996; Kelsey, Lindner & Dooley, 2002; Moore 
& Kearsley, 2005; Oliver & Shapiro, 1993; Torkzadeh & Van Dyke, 2002). 

Other studies found that the effectiveness of e-leaming is dependent on e-leaming environmental 
characteristics (Song Sang-ho, 2004). They focus mainly on design and operational elements of e-leaming, and 
identify elements including composition of learning contents, learner support, convenience or ease of use, 
interaction such as teachers’ feedback and operators’ social role, user interface, teachers’ capability, various 
evaluation methods, preliminary education, interaction between learners, accessibility, and physical and 
psychological support (Park Seong-ik, Yoon Soon-Gyeong, 2000; Seo Shin-seok, 2003; Seo Hye-jeon, 2001, Lee Ji- 
yeon, 2004; Cho Il-hyeon, Lim Gyu-yeon, 2002; Choi Jeong-im, 1999; Harasim, 1986; Kelsey, Lindner & Dooley, 
2002; Thurmond et al., 2002). 



Purpose of the Paper 

Although there are a number of theories and studies on the effectiveness of e-leaming, few studies take a 
comprehensive look at various e-learning factors such as learners’ profile, general learning-related elements, web- 
based learning-related elements, learners’ satisfaction, learners’ academic achievement, and learners’ participation. 
As a result, Lee In-sook (2003) and Yoo Pyeong-jun (2003b) strongly raised the need to conduct comprehensive 
research taking all e-learning factors into account. This study aims to design a comprehensive theoretical model to 
explain the effectiveness of e-learning based on existing studies, which revealed the effectiveness of e-learning 
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according to learners’ characteristics and environmental characteristics, and to verify the model by measuring paths 
between factors and effect size to identify factors contributing to the perceived effects of e-leaming. This will help 
us understand the elements that must be considered to enhance the effectiveness of e-learning. 

Research Questions 

The purposes of this study are to design a model that can explain factors affecting the perceived effects of 
e-learning; to identify a structural relationship between those factors; and to assess the model fit of the structural 
equation model. To that end, it has set up the following research tasks. 

Research Task I : To design a theoretical structural model of factors influencing the effects of e-leaming 
perceived by elementary school students based on existing studies 

Research Task II : To devise a provisional stmctural equation model based on the theoretical model, and 
test the model fit of the equation model 

Research Task II -1: To decide whether the provisional stmctural equation model based on the theoretical 
model is the optimum model or not, and, if not, to identify the optimum model 

Research Task II -2: To discover whether hypothetical paths between factors affecting the perceived effects 
of e-learning are significant 

The Model 

This study used the Input-Environment-Outcome (I-E-O) model by Astin (1993) as the conceptual basis for 
a stmctural model of factors influencing the perceived effects of e-learning. The I-E-0 model has been used by 
many researchers to verify the relationship between learner’s characteristics, e-leaming environmental 
characteristics, and e-leaming effectiveness (Astin & Sax, 1998; Campbell & Blakey, 1996; House, 1999; Kelly, 
1996; Knight, 1994a, 1994b; Long, 1993; Pace, 1976). 

Adopting the ideas of Astin (1993), the study defined “input” as the characteristics of individual learners 
participating in educational programs and “environment” as the practical experiences and activities educational 
programs provide for students. Based on existing studies, “outcome” was defined as the learners’ academic 
achievement and satisfaction through participation in learning (Kang Yeoung-hwan, 2004; Koo Gyo-jeong, 2005; 
Seo Hye-jeon, 2001; Yoo Pyeong-jun, 2003b; Lim Jeong-hoon, Lee Hang-nyeong, 2003). 
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Figure 1 shows the theoretical structural model indicating the elements of each factor and hypothetical 
paths between them. 



E-learning environmental 



Learners' characteristics 



E-learning effectiveness 



characteristics (Environment) (Input) (Outcome) 




Measurement Issues 

Table 1 shows the composition and sources of a survey tool to analyze a structural model. The 
questionnaire has three factor categories - e-learning environmental characteristics, learners’ characteristics, and e- 
leaming effectiveness. The e-leaming environmental characteristics category consists of four parts, which as are the 
same as the four elements - composition of learning contents, learner support, usability, and interaction. The 
learners’ characteristics category also consists of four parts - intrinsic motivation, extrinsic motivation, academic 
self-efficacy, and computer self-efficacy - and the e-learning effectiveness category three parts - learners’ academic 
achievement, learners’ participation, and learners’ satisfaction. 
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Table 1. Composition and Sources of Survey Tool 



Category 


Elements 


Sources 


E-learning 

environmental 

characteristics 


Composition of learning contents 


Seo Hye-jeon (2001) 


Usability 


Yoo 11, Hwang Jun-ha (2002) 


Learner support 


Stewart, Hong & Strudler (2004) 


Interaction 


Learners’ 

characteristics 


Intrinsic motivation 


Velayo (1993); Stein (1997); 

Chung Jae-sam, Lim Gyu-yeon (2000) 


Extrinsic motivation 


Academic self-efficacy 


Yang Myeong-hee (2000), Lee Jae-gyeong (2000) 


Computer self-efficacy 


Al-Khaldi & Al-Jabri (1998); Oh Yoon-jin (1999) 


E-leaming 

effectiveness 


Learners’ academic achievement 


Lim Jeong-hoon, Chung In-seong (1999); Seo Hye- 
jeon (2001) 


Learners’ participation 


Yoo Pyeong-jun (2003b); Chung Jae-sam, -Lim Gyu- 
yeon (2000) 


Learners’ satisfaction 


Lim Jeong-hoon, Chung In-seong (1999); Seo Hye- 
jeon (2001) 



The questionnaire is composed of 53 questions measured 1 to 5 according to a Likert-type scale, with 1 
meaning “strongly disagree”; 2 “disagree”; 3 “neither agree nor disagree; 4 “agree”; and 5 “strongly agree.” The 
questions were created based on existing studies to verify a provisional structural model devised on the basis of a 
theoretical structural model. In relation to the question items, a validity test by experts and a reliability test were 
conducted. For the reliability test, a preliminary survey was also carried out on 130 sixth-graders in elementary 
schools in Seoul from Apr. 5-7, 2006, to compute Cronbach’s a of the questionnaire. The reliability of the 
questionnaire was .94, which is fairly high. 



Data Analysis 

The survey for the study was conducted on 1,500 sixth-graders in seven elementary schools in Seoul and 
Gyeonggi Province Apr. 18-29, 2006. Some 82.7 percent, or 1,241, out of the 1,500 questionnaires distributed were 
collected, and 1,154 of them were found valid for statistical analysis. 

With the collected data, a confirmatory factor analysis of a measurement model based on the theoretical 
structural model was carried out using AMOS 5.0. The analysis included the assessments of the model fit index and 
construct reliability, and calculation of the variance extracted index to evaluate if the scales of the measurement 
model are representative of the factors concerned. A discriminant validity test was also conducted using a 
correlation matrix analysis of constructs. An analysis of a measurement structural equation model based on the 
measurement model showed that AGFI, a model fit index, was not high enough to be valid, and as a result, six 
measurement variables were removed using squared multiple correlations, modifying the measurement model. A 
model fit test of the modified measurement model found that most of the model fit indices improved in the 
measurement structural equation model with the value of / 2 =2,401.83, df=979, GFI=0.913, AGFI=0.900, 
TLI=0.938, CFI=0.943, and RMSEA=0.036. The reliability and validity of the measurement model were also 
assessed to be valid. 

All the results of the confirmatory factor analysis were valid enough to next conduct an analysis of a 
structural equation model, which was carried out for a provisional model based on the measurement model 
modified. A model fit test of the provisional model showed a fairly good fit, but the values of GFI and AGFI were 
lower than acceptable, raising the need for a better model than the provisional model. After checking the 
modification index of the provisional model and the theoretical background of existing studies, two paths indicating 
a direct relationship between interaction and learners’ participation, and between composition of learning contents 
and learners’ participation, were added. 
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Table 2. Results of Model fit Test of Provisional Model and Modified Model 



Model 


x 2 


d.f 


P 


V/df 


GFI 


AGFI 


TLI 


CFI 


RMSEA 


Provisional model (1) 


2,949.59 


1,001 


0.000 


2.947 


0.895 


0.882 


0.916 


0.923 


0.041 


Modified model (2)* 


2,776.39 


999 


0.000 


2.779 


0.900 


0.887 


0.924 


0.929 


0.039 


Model 1: Model 2 


173.20 


2 


0.000 


- 


- 


- 


- 


- 


- 



Modified model = Provisional model + (Interaction — ► Learners’ participation, Composition of learning contents —> Learners’ 
participation) 



A model fit test of the modified provisional model showed an improvement of model fit with the value of 
/=2, 776.39 (p=0.000), df= 999, //df=2.779, GFI=0.900, AGFI=0.887, TLI=0.924, CFI=0.929, and 
RMSEA=0.039. Thus, the modified model was chosen as the final model. Figure 2 shows the final modified model 
and path coefficients. 




Efforts to raise the effectiveness of e-leaming are needed not only in elementary and secondary education 
but also in university and corporate education. Many studies have therefore been conducted to assess the 
effectiveness of e-leaming and thus provide an effective e-leaming service. This study aims to design a model 
explaining the effects of e-learning perceived by elementary school students and assess the paths and effect size of 
various factors influencing the effectiveness of e-leaming. This is expected to promote in-depth understanding of 
how elementary school students learn in e-leaming and enable creation of an e-leaming service that suits the needs 
of child learners. 

The following conclusions can be reached based on the study results. 

First, the final stmctural equation model was proved valid to explain factors influencing the effects of e- 
leaming perceived by elementary school students through the confirmatory factor analysis and stmctural equation 
model. The model fit test of the final model showed a fairly good fit with the value of x 2 =2,776.39(p=0.000), 
df=999, //df=2.779, GFI=0.900, TFI=0.924, CFI=0.929, and RMSEA=0.039. This means that the final stmctural 
equation model is a proper stmctural model to explain the effects of e-leaming perceived by elementary school 
students. 
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Second, unlike models in past studies, the model in this study showed that e-leaming environmental 
characteristics influence the effectiveness of e-leaming through the mediation of learners’ characteristics. For 
example, in the composition of learning contents, an element of the e-leaming environmental characteristics that 
affect academic achievement, the effect size was significant at .447, even when taking into account indirect effects 
alone that take place through the mediation of intrinsic motivation, computer self-efficacy, and learner’s 
participation. Learner support also showed a significant effect size of .228 toward academic achievement, even 
taking into account indirect effects alone that took place through the mediation of intrinsic motivation and computer 
self-efficacy. These results indicated that e-learning environmental characteristics can have indirectly influence the 
effectiveness of e-leaming through the mediation of learners’ characteristics. This study is distinguished from past 
studies by its interest in the mediating effects of e-learning environmental characteristics and learners’ 
characteristics on the effectiveness of e-leaming (Seo Hye-jeon, 2001; Koo Gyo-jeong, 2005; Chung Jae-sam, Lim 
Gyu-yeon, 2000). 

Third, it was found that e-leaming environmental characteristics have positive effects on learners’ 
characteristics. In the case of intrinsic motivation, learner support, composition of learning contents, and usability 
had positive effects in that order, with the effect size being .475, .378, and .207, respectively. Learner support (.319) 
had a positive effect on extrinsic motivation. On academic self-efficacy learner support (.641) had a positive effect, 
and on computer self-efficacy learner support (.907) and usability (.121) had a positive effect. This showed that e- 
leaming characteristics influence the effectiveness of e-learning in combination with specific learners’ 
characteristics. 

Fourth, an analysis of learners’ characteristics influencing the effectiveness of e-learning found that 
extrinsic motivation and academic self-efficacy do not have a significant effect on learner’s academic achievement, 
participation, or satisfaction. This goes against existing study results: Wlodkowski (1999) and Chung Jae-sam and 
Lim Gyu-yeon (2000) said that extrinsic motivation had a direct impact on learners’ academic achievement and 
participation in e-leaming, and Lee In-sook (2002) found that academic self-efficacy was a major factor affecting 
learners’ academic achievement. The results suggest that the stmctural model could include another element of e- 
leaming effectiveness on which extrinsic motivation and academic self-efficacy have indirect impact, or a new path. 

Fifth, the effect size of each factor influencing the effectiveness of e-learning can be compared by 
analyzing indirect, direct, and total effect size. Of factors affecting learners’ academic achievement, intrinsic 
motivation had the biggest effect size with .919. Composition of learning contents came second with .447, followed 
by learners’ satisfaction (.346), and learner support (.228). In relation to learners’ participation, composition of 
learning contents showed the biggest effect size with .267, followed by interaction with .203. In the case of learners’ 
satisfaction, intrinsic motivation was found to have the biggest impact with the effect size being .879. Composition 
of learning contents came in second with .439 followed by learners’ participation (.179) and learner support (.160). 
This suggests that to raise the effectiveness of e-learning, that is, learners’ academic achievement, participation, and 
satisfaction, it is essential to actively work on development of factors found to affect it most and take child learners 
into account when doing so. 



Future Research 

Based on the study results, some suggestions can be made in relation to future study on the effectiveness of 
e-leaming. First, the study showed that it is intrinsic motivation, a learner’s characteristic, that has the greatest 
influence on learners’ academic achievement. This indicates that when developing a new e-learning service for child 
learners, the focus in a new service needs to be placed on ways to increase intrinsic motivation, which makes 
learners feel rewarded and pleased with their learning activity. This is in contrast to existing services, which focus 
on ways to boost extrinsic motivation, like providing cyber prize money, prizes, gifts, and vouchers. 

Second, as mentioned earlier, the e-leaming service for elementary and secondary school students needs to 
be more carefully thought out and effective than that for university students and corporate workers because children 
are less capable of regulating their own study than adults (Song Sang-ho et al., 2005). This raises the need for 
serious study on the differences in characteristics between child and adult learners. This study produced some results 
that go against past studies. Previous studies, in many cases, have focused on adult learners and thus failed to reflect 
the characteristics of child learners in e-leaming. Many other studies on ways to enhance the effectiveness of e- 
leaming have also failed to present the differences in e-leaming environments for children and adults. Therefore, as 
Kim Do-il et al. suggested (2003), it is necessary to work to increase in-depth understanding of the e-leaming 
environment optimized to suit the level of child learners, who analyze, use, and understand information in a different 
way from adults due to difference in cognitive ability. 
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Concluding Comments 

E-leaming is a comprehensive cyber learning space where various environmental factors and factors related 
to learners’ characteristics play a major role. As such it is essential to identify those various factors and study in 
depth what role they play in raising the effectiveness of e-leaming. Existing studies on the effectiveness of e- 
leaming have usually applied one or two factors for evidence, but future studies should look at all e-learning 
composition factors in a comprehensive and complete way. While it is important to study how each e-leaming factor 
influences the effectiveness of e-leaming, it is more important to establish a systematic relationship between e- 
leaming factors, and discuss ways to operate e-learning effectively and make the e-leaming environment more 
learner-friendly. 
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A Study on the Improvement of Reading Comprehension through Visual 
Organizers in an Online Reading Course 

Myeonghwa Park, Sung-hee Jin, Eunmo Sung 
(Seoul National University, Korea) 

Introduction 

Various kinds of reading strategies have been applied in designing e-leaming reading contents for improving 
reading comprehension. Presenting structured overview in pre-reading phase (Barron, 1969; Griffin & Tulbert, 
1995), embedding questioning signals (Hartley, Tureman, & Pigram, 1984; Nist, & Hogrebe, 1984) and highlighting 
key sentences or phrases (Lee, 1998; Dyson & Gregory, 2002; Mok, 2004; Sohn, 2004) in the middle of reading 
phase, and using graphic organizers and summarizing in post-reading phase have effect on reading comprehension 
(Dunston, 1992; Bae, 2004). 

In supporting recent stream of research in text design, more specifically text structure, US National Reading Panel 
Report (2000) emphasize the significance of text structure strategies and suggest that for adults’ reading and 
teenagers’ reading, comprehending topics and inferring is more rather than on learning vocabulary or word-relating. 

As commonly known, text structure is of significance shown well in the expository paragraphs, beginning with a 
topic statement, and then, by several subdivision to support and connect the central idea. In expository text the 
information is organized in a logical manner using various expository text structures and they include description, 
enumerative or listing, sequence, comparison and contrast, cause and effect and problem and solution (Gillet and 
Temple 1994; Irwin 1991; Meyer & Freedle, 1984). 

According to Block’s investigation (1986), low comprehending readers tend to read in reflective mode, directing 
their attention away from the text and toward themselves, and focusing on their own thoughts and feelings rather 
than on the information in the text. Whereas high comprehending readers’ response in extensive mode to deal with 
the message conveyed by the author, focusing understanding the author’s ideas, and not to relate the text to 
themselves affectively or personally. 

Regardless of using the strategy in pre-, post-reading or in the middle of reading, teaching text structure impact on 
comprehension (Pehrsson and Denner 1988; Englert 1990; Mosenthal, 1994). In order for a reader to integrate 
information presented to the main idea consistently, text structure needs to be reminded, and at this point, a tool or 
opportunity reminding the text structure will surely have effect on the reading comprehension in a positive manner. 
Visual Knowledge Organize^ VKO), for instance, is a strategy representing the organization of information and 
knowledge from the texts in a visual format (Rha & Ahn, 2006). Visual knowledge organizer including structured 
overview, concept map, and graphic organizer has revealed its effectiveness in recent research (Dunston, 1992; 
Moore & Readence, 1984). In addition, visual knowledge organizer is expected to operate as a cognitive tool to 
maintain reading in extensive mode (Block, 1986). The purpose of the current study is to develop visual knowledge 
organizer and to examine its effects for improving reading comprehension in online reading course. 

Visual Knowledge Organizer 

As an explicit technique for structuring the text, VKO is a strategy representing the organization of information and 
knowledge from the texts in a visual format (Rha & Ahn, 2006). Human beings tend to encode information in a 
sequence and then save it. However, when several texts are delivered, the linear process of information cause 
cognitive overload and bring about difficulties in decoding and saving (Sweller, 1994; Paivio, 1986). According to 
dual code theory, human beings enable to treat visual information in parallel, saving and decoding text information 
separate. From this point, VKO has been developed on the ground of parallel processing of visual stimulus coping 
with the linear processing of text information. VKO is also a cognitive strategy utilizing visual aids for improving 
efficiency and effect of text comprehension (Rha, Ahn, 2006). VKO is expected to function as a visual cognitive 
tool based on the visual intelligence that humans obtain. Some features that VKO evolve are as follows: i) VKO 
presents the key information of the text in a visual format in order to provide structural knowledge (DiCecco, 1992; 
Pehrsson & Denner, 1989). ii) VKO represents overview on the text by providing semantic text information, iii) 
VKO facilitate to memorize key words in text information by supporting retrieval cues from long term memory 
(Levin, 1987). 
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In the context of reading expository text, VKO facilitate to improve text comprehension by providing visually or 
mentally represent of text structure. Text structure can be represented in the format of verbal and visual. However, 
visual representing strategies of text structure are needed to efficiently, effectively process and memorize a good 
amount of text information. In other words, representing VKO about text structure will improve learner’s reading 
comprehension. 

Design & Development of VKO 

On the basis of the literature review and the results of survey targeting experts, the optimal VKOs depending on 
text structure patterns were developed for improving reading comprehension in e-learning environment. The types of 
VKO and design principles were driven through literature review. In case of expository text, the text structure 
patterns commonly include: (1) sequence, (2) comparison/contrast, (3) description, (4) problem/solution, and (5) 
cause and effect/process. 

Because text structure is a kind of semantic concept, text structure could be represented in the various types of 
VKO (Gillet and Temple 1994; Irwin 1991). Therefore, optimal VKO needs to be represented, which enables 
learners to perceive the semantic structure of text with ease. The design principles for VKO has been established 
such as symbolicity, simplicity (Rha & Han, 2006; Jo et el., 2005; Thompson, 1994), and predictability (Rha & Han, 
2006) through literature review. The validity of VKO was examined by 6 subject matter experts (SME) in the 
domain of teaching English. And thus, for applying VKO into e-leaming contents, VKO should reflect the 
characteristics of e-learning environments in terms of improving the learning achievement. For this, the 
development principles of VKO has been established such as interactivity (Baek, 1995; Lim, 1999; Morris & 
Hinrichs, 1996), navigability (Han, 2006), and simultaneity (Morris & Hinrichs, 1996). Each of detailed principle is 
as follows. 

Design Principles for the Conceptual Models of VKO 

Symbolicity 

VKO needs to be designed in a visual format which enables to represent the semantic text structure. From the 
semiotic perspective, VKO can be regarded as a ‘sign’, composed a ‘signifier’ and ‘signified’. A signifier represents 
figures of a specific object or a concept like pictures, languages, sounds, and so on. A signified implies the meaning 
or concept that a signifier possesses. Seeing VKO as a sign, the figure represented visually is a signifier and the 
underlying semantic structure of the signifier is a signified. That is, the principle of symbolicity is to design VKO in 
compliance with signifier and signified. 

Simplicity 

VKO needs to be designed in the simplest form. Text structure can be represented in a variety of visual forms. In 
order for learners to perceive the holistic information on text structure at a glance, VKO should be composed of 
minimum information which require for comprehending text structure without redundant information. Jo et al. 
(2005) suggested simplicity as a design principle of e-leaming contents. 

Predictability 

VKO needs to be designed for learners to predict what VKO means and which functions VKO has (Rha & Han, 
2006). Human beings have the information on images by means of repeated experiences and learning. VKO should 
be designed reflecting conceptual mental model on this visual information. That is, VKO is a strategy of organizing 
text, which cannot be an object to demand learning, must be comprehended by every learner through their prior 
experiences. 

Development Principles for Applying VKO to Online Reading Contents 

Interactivity 

VKO needs to be designed so that learner may efficiently obtain structural text information. To interact between 
learner and VKO, it will be considered an immediate reaction, learners’ adaptability, convenience, and usability of 
learning (Son, & Jeon, 2002). An immediate reaction means that learner instantly recognize how to use a VKO. 
Learners’ adaptability and convenience means that learner directly utilize VKO. Usability of learning means that 
learners should be received substantial helps in read texts by using VKO. 

Based on this, VKO was developed in three phases considering the levels of interaction. Phase I was developed in 
the form of completed VKO including text stmcture and details. Phase 2 was developed in the form of half- 
completed VKO including text stmcture and partial blanks for the details. Last phase 3 was developed in the form of 
open-ended VKO, which learners constmct by themselves. Learning activity to constmct visual knowledge through 
reading can guide meaningful learning on the ground of self-directed learning. The Phases applied to VKO appear, 
in Table 1 . 
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<Table 1> The Phases applied to Visual Knowledge Organizers 




Navigability 

VKO needs to be developed functioning as navigation in order for learners to comprehend easily verbal and visual 
information representing text structure. When a part of VKO is clicked, the relevant contents are highlighted. This 
function make VKO to play a role of navigational aid exploring the holistic structure and contents of text. 

Simultaneity 

VKO needs to be presented with text simultaneously. e-Learning contents can be developed dividing the frames 
even though there are some differences depending on developed programs. In case of developing HTML based e- 
learning contents, VKO can be presented with text simultaneously. If VKO is presented in another window, pop-up 
window can disturb learners’ reading activities and there are some difficulties for learners to use VKO. Thus, VKO 
is designed in divided frame to use it as learning support aid and to make learners pay attention to reading activities. 
The optimal representation forms of VKO driven through the results of survey and the examples of developed e- 
leaming contents are shown in table 2. 
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<Table 2> The representation forms of VKO and the 
examples of developed e-leaming contents 



Text Structure 



Visual 

Representation 



Examples of Contents 



Chronological 

sequence 



Description 



Comparison/ 

Contrast 



Problem/ 

Solution 



Process/Cause 
and effect 



of VKO 




Methodology 

The purpose of current study is to examine the effectiveness of visual knowledge organizers on reading 
comprehension in online reading course. The overview of research methods are as follows. 

Participants 

Visual knowledge organizers were adapted to develop the contents for online course of English reading 
comprehension in the S cyber university. The participants were 54 students that took the online reading course, who 
were seniors (male: 20, female: 34). They are majoring in English and have taken online courses for approximately 
3 or 4 years. They already learned the text structure, however, haven’t experienced in taking online English reading 
comprehension course to utilize VKO. And thus, the participants were appropriate subject for examine the effects of 
VKO. 

Research Design 

The comparison on the results of pretest and posttest related English reading comprehension were analyzed. In this 
study, independent variable is the usage of VKO and dependent variables are reading comprehension and 
satisfaction on learning. English reading comprehension was categorized into comprehending topic, comprehending 
the details, inferring, understanding the text structure. This study was analyzed how much VKO affected on those 
specific English reading comprehension. 

Materials 

The online instrument for reading comprehension test comprises of 1 5 question items with three different kinds of 
expository texts. The difficulty of the test was intermediate and high intermediate with approximately 600-800 
words per paragraph. Both pre-, and post test are composed of 18 question items with subordinate category of 
comprehension asking: four question items for main theme or topic, four for understanding details, four for 
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inferring, and four text structure. All question items are multiple choices and conducted online with limited time of 
thirty minutes. The reliability was cronbach a=. 82. 

The test instrument for examining students’ satisfaction on VKO use was developed on the basis of the literature 
review. It consisted of 10 items including/using 1-5 likert scales. One of doctor and three of graduate candidates 
majoring in Educational Technology reviewed the instrument and examined the degree of credibility. The degree of 
credibility was a=.19. 

Procedures 

This study was conducted through the process of design, development, implementation, and evaluation. First of all, 
principles for designing and developing VKO according to the types of text structures were driven and the contents 
for online English reading comprehension were developed in the phase of design and development. Students had 
taken the course, learned with using VKO for 15 weeks from February to June, 2007 in the implementation phase. 
After finishing the course, quantitative and qualitative date were collected and analyzed such as the result of reading 
comprehension, satisfaction of learning with using VKO, and focus group interview. 

Data Analysis 

For examining the effectiveness of VKO, two-dependent samples t-test between the results of pretest and posttest 
was examined. In order to examine the effectiveness of VKO on comprehending topic, comprehending the details, 
inferring, and understanding the text structure that are subordinate reading comprehensibility, two-dependent 
samples t-test between the results of pretest and posttest on subordinate reading comprehensibility. The program, 
Window SPSS 12.0, was used to analyze the data. 

Further more, Effect size was analyzed in order to examine the practical significance of effects. Survey data and 
interview data, and thus, were analyzed to examine students’ satisfaction on VKO. 

Results 



Effect of VKO improving reading comprehension 

The result of t-test between pretest and posttest is displayed in Table 1. Pretest was Mean=33.10, SD=19.29, and 
posttest was M=54.93, SD=19.76. There was a significant difference t = -10.374, p<.05. In other words, the result 
indicated that using a VKO in online reading course was proved to have positive effect in improving reading 
comprehension. 

<Table 3 > Analysis of t-test about reading com prehension 





N 


Mean 


Standard 
_ . . t 

Deviation 


Sig. 


Effect 


Pre- 


Score 


size 




19.29 




54 


33.10 






test 

Post- 






10.374 


.000 


1.104 


54 


54.93 


19.76 






test 













The result reported there is a statistically significant difference. In real world, however, using VKO may be 
practically non-significance in improving reading comprehension, For that reason, we conducted an analysis about 
the effect size of VKO. Effect sizes can be generally interpreted as fellow: small is below 0.2, medium is 0.5, and 
large is great than 0.8(Cohen, 1988). The result of effect size was A=1.104. Therefore, Using a VKO in English 
reading would predict a practical effect in improving reading comprehension. 

Effect of VKO in sub-ordinate reading comprehension 

VKO in e-Leaming was reported to improve learner’s reading comprehension in general, we further analyzed what 
effect VKO have on sub-ordinate reading comprehension; understanding main idea, details, text structure, and 
inferring, This analysis was undertaken by t-test between pretest and posttest score of sub-ordinate reading 
comprehension. After that, Effect sizes were computed for four parts. Table 2 presents the means and standard 
deviations, t-value, significant, and effect sizes. 
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<Table 2. > Analysis on t-test of sub-ordinate reading comprehension 



Standard 

Compre ^ Means Deviatio 

hension 

n 



Sig. 



Effect 

Sizes 



Pre 54 7.87 8.51 

Topic 

Post 54 14.19 7.55 



Pre 54 18.98 11.05 

Detail 

Post 54 13.20 7.49 



Pre 54 3.93 5.86 

Infer 

Post 54 17.03 8.75 



Struc- Pre 54 2.31 4.90 

tore p ost 54 10.74 6.26 



-5.39 .000 .837 



3.97 .000 -.771 



-10.92 .000 1.528 



-8.17 .000 1.346 



The results of t-test indicated a significant difference in four parts; topic = -5.59, detail = 3.97, infer = -10.92, 
structure = -8.17 at the level of p<.05. This result interpreted that using a VKO also has a positive effective in sub- 
ordinate reading comprehension. Effect sizes were different among four parts. It was a practically significant in 
main idea (A=1.104), inferring (A=1.528), and text structure (A=1.346). However, it was a practically non- 
significance in details. 

The result was interpreted that using VKO influenced understanding of macro structure, main topic, and inference 
of the text. However, it seemed that VKO have near to never effect on details. In spite of a practically non- 
significance on details, reading comprehension was presented statistically significant. It may be possible to state that 
reading comprehension due to cognitive processing of human being that sub-capability in reading comprehension 
was interacted as for dependent factor rather than independent one. 



Satisfaction with using VKO 

We conducted FGI (Focus Group Interview) to identify satisfaction with using VKO in online English reading 
comprehension course. FGI was conducted with six participants who were selected randomly out of the whole 
participants. Table 4 is a questionnaire, which used in FGI. 

Table 3. Questionnaire for Focus Group Interview 





Questionnaire 


1 


Does VKO help with learning? 


2 


Does VKO help with understanding text 
contents? 


3 


Does VKO help with improving 
vocabulary? 


4 


Does VKO help with grammar based 
reading? 


5 


Does VKO help with understanding main 
idea? 


6 


Does VKO help with inferring or guessing 
the unknown words? 


7 


Does VKO help with analyzing of text 
structure? 


8 


Do you want to use a VKO when you read 
next time? 



According to the result of FGI, learners reported that using VKO in reading comprehension supported facilitating 
of understanding main idea and analyzing text structure in macro level. Also, they stated to use a VKO continuously 
when they read. On the contrary, learners thought that VKO was not helpful for understanding meaning of difficult 
words and grammar based on reading. This result of interview is matched the result of t-test between pretest and 
posttest in reading comprehension 

Consequently, it seemed that using VKO in online learning of reading comprehension was effected understanding 
text. 
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Conclusion 



This study was undertaken to develop and to identify effect of VKO, which was to improve reading comprehension 
in online reading comprehension course. VKO was designed and developed by means of principles such as clarity, 
simplicity, predictability, interactivity, navigability, simultaneity. 

To identify the effect of VKO, Quantitative and qualitative measures were used: achievement score of reading 
comprehension (pretest, posttest score), learning satisfaction through focus group interview. Following the result, 
using a VKO in online reading comprehension course was identified positive effect in effectiveness and satisfaction 
of learning. VKO was facilitated understanding main idea, macro text structure, and inferring. However, it was 
practically non-effect about details. 

Finally, the developed VKO in this study strongly suggest that it was more effectiveness not understanding micro 
text information, but understanding macro text information such as main idea, topic, and text structure at the macro 
level. 
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Abstract 

This study sought to evaluate the effectiveness of a peer feedback strategy in asynchronous online 
discussions. Specifically, this exploratory study examined the impact of peer feedback in online discussions 
on students' perceived and actual critical thinking skills in terms of receiving and providing peer feedback 
Participant interviews and pre and post surveys targeting critical thinking skills were utilized in a mixed 
method study approach. Results indicate that participants perceived the peer feedback strategy as having 
impacted their learning at a higher cognitive level. Qualitative data reveals how the peer feedback process 
impacted students' learning, both as receivers and providers of peer feedback. The study has implications 
for teaching and designing online courses that employ asynchronous discussions. 



Background 

The increase of online learning, including fully online and hybrid courses continues to increase with no 
plateau sight. In the United States alone there were nearly 3.2 million students taking at least one online course as of 
fall 2005 (Allen & Seaman, 2006). Within these courses online discussions and dialogue have been heralded as a 
potentially powerful tool that can assist students in the construction of knowledge, serve as a scaffold that allows for 
multiple perspectives resulting in additional knowledge, negotiation of meaning, and an understanding of knowledge 
gaps a learner may possess (Land, Choi & Ge, in press; Haavind, 2006; Maor, 2003; Roehler & Cantlon, 1997). This 
method aligns with social constructivist theories, specifically Vygotsky's social constructivist theory of learning 
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(Driscoll, 2004). Social constructivist theory argues that learning is social in nature, suggesting that online 
discussions can become fruitful learning environments when they are developed in an appropriate manner. In 
particular, the use of online discussions can be tied to Vyogtosky's notion of the Zone of Proximal Development 
(ZPD), the distance between what learners can do by themselves and what they can achieve with assistance (1978). 

Peer Feedback in Online Learning 

Feedback, as an integral aspect of the learning process, is promoted by researchers and practitioners alike 
(Mory, 2004; Driscoll, 2004). In his analysis of feedback research in learning, Mory (2004) suggested that feedback 
is a "critical function in knowledge acquisition" (p. 777) and that without a feedback mechanism learning would not 
occur. Whether learners are in an online or traditional environment this holds true, and as Dunlap (2005) suggested 
feedback for online learners can also serve to counter feelings of disconnect or isolation while a lack of feedback can 
slow learners’ progress. Nicol and Macfarlane-Dick (2006) suggested that feedback serves as a form of formative 
assessment, designed to improve and accelerate learning. Specifically, feedback is defined "... as anything that 
might strengthen the students’ capacity to self-regulate their own performances" (p. 206). 

Given this, proponents of online learning environments have several reasons to explore feedback strategies, 
including peer feedback. Instructor workload, for one, has been an issue discussed, though mostly anecdotally, in the 
literature (Dunlap, 2005; Dennen, 2003). Just as students perceive their workload increasing for online courses, so 
do instructors. Peer feedback has the potential to reduce an instructors’ workload while simultaneously providing the 
type of feedback that is important to students’ learning and growth. 

Boud, Cohen, and Sampson (1999) explained the importance of peer learning or feedback in today’s world, 
including employers’ demands that college graduates possess a broader range of skills including the ability to 
communicate effectively. Specifically, they cite the need for transferable skills, key competencies, and generic 
attributes or competencies, all skills that foster lifelong learning. Furthermore, peer learning and lifelong learning 
skills both incorporate "the development of learning outcomes related to collaboration, teamwork, becoming a 
member of a learning community, critical enquiry and reflection, communication skills, and learning to learn” (p. 
417). 

Liu, Lin, Chiu and Yuan (2001) described peer learning as "valuing the exchanges of critical feedback 
among peers and modifying works according to peer feedback” (p. 246). They suggested that peer review or 
feedback can foster an authentic learning environment in which students actively construct knowledge. In their study 
of a web-based peer review system they described what students achieved through the peer review process: peer 
review allowed them to "read, compare, or question ideas, suggest modification, or even reflect how well one’s own 
work is compared with others. While processing these cognitive functions one monitors the adequacy of their work” 
(p. 248). It is during this process that cognitive functions, including critical thinking, are hypothesized to increase. 
Correspondingly, Boud et al. (2001) expressed the idea that students learn a great deal by explaining their ideas to 
others and by participating in activities in which they can learn from peers. 

Just as there are many advantages to using peer feedback, there are also potential challenges. The first 
challenge that instructors need to be aware of is that many students are not familiar with the peer assessment 
process, not in any formalized way; as Palloff and Pratt explain, meaningful peer assessment "is not a naturally 
acquired skill" (1999, p. 123). Topping, in his review of peer assessment, noted several of the more prominent 
challenges including: poor performers not accepting peer feedback as accurate, students refusing to take 
responsibility for assessing peers if they see the peer assessment as substituting for instructor feedback or if they are 
part of a small "socially cohesive" group, a need to monitor peer assessment for potential abuse of peer power, and 
a concern about issues of reliability and validity (Topping, 1998, p.256). 

Teaching and Assessing Critical Thinking 

There is much talk given to the concept of critical thinking and related skills in learning and instruction, 
and as many definitions and perspectives as there are disciplines. Facione and Facione (2007) define critical thinking 
as “reflective decision-making and thoughtful problem solving about what to believe and do” (p. 44). Similarly, 
Halpem (2003) defines critical thinking as “cognitive skills and strategies that increase the likelihood of a desired 
outcome. . . thinking that is purposeful, reasoned, and goal-directed — the kind of thinking involved in solving 
problems, formulating inferences, calculating likelihoods, and make decisions” (2003, p. 6). 

The many approaches to teaching critical thinking skills is parallel to the many approaches to assessing 
them as well. Spicer and Hanks (1995) reported on seven standardized critical thinking tests available as well as 
several performance assessment approaches that can be used as outcome measures within various subjects. 
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Standardized tests can provide useful information that is diagnostic and may help to guide instruction. However, 
multiple measures of critical thinking should be used whenever possible, since critical thinking is not a general 
ability but rather a complex set of general and specific factors. For example, in addition to multiple-choice tests, 
open-ended assessments and interviews could be incorporated into assessments of critical thought. 

Studies have shown that online discussions can support critical thinking (Gunawardena, Lowe, and 
Anderson, 1997; Bullen, 1997; and Jeong, 2000 as cited in Yang, 2002). Furthermore, several recent studies have 
reported on the assessment of critical thinking skills focusing on students engaged in online discussions. The 
Community of Inquiry (COI) Model is a framework that was developed to assess the nature of critical thinking, 
specifically higher-order thinking, in online discussions from a practical approach (Garrison, Anderson & Archer, 
2001). The COI is a four phase model (triggering, exploration, integration and resolution), based on earlier work 
(see Garrison & Archer, 2000), that was developed by conducting a content analysis of transcripts form online 
discussions. The study concluded that the framework could provide instructors with a tool to asses critical thinking 
but also cautioned that in order to have higher order thinking there must be sustained critical discourse, one that 
involves the full cycle of the critical thinking process, a skilled instructor facilitation, and interaction that is both 
coordinated and synergistic (p. 11). 

Similarly, Hemphill and Hemphill (2006) investigated the affect of virtual guest speakers in an online 
educational technology graduate course and the extent to which the students engaged in critical thinking within 
online discussions. The results indicated that higher-order thinking occurred throughout the course, and reported that 
their findings had implications for instructors, including monitor discussions and a "judicious use of appropriate 
responses by facilitators [to] maintain a high frequency and critical thinking level of responding by the students "(p. 
292). 

Purpose of the Study 

Prior to this study, the peer feedback strategy was implemented in a graduate level technology integration 
course in the spring semester of 2005. Feedback from students about ease of use and perceptions of the process 
informed the use of peer feedback in this study (See Ertmer, Richardson, Belland, Camin, Connolly, Coutlhard, Lei 
& Mong, 2007). The peer feedback instructional strategy was initially envisioned as a way to provide ongoing 
feedback to students while decreasing instructor load in online courses. This study was designed to evaluate the 
effectiveness of a peer feedback strategy in asynchronous online discussions. Specifically, this study examined the 
impact of peer feedback in online discussions on students' perceived and actual critical thinking skills. The research 
questions included: 

1 . What were students’ perceptions of the impact of the peer feedback strategy on their learning? in 
terms of providing and receiving peer feedback? 

2. What was the effect of peer feedback on students' critical thinking skills, as measured by a critical 
thinking skills test? 



Methods 

Given that this is an exploratory study, a mixed model research approach was utilized (Johnson & 
Christensen, 2004). The mixed methods research design maximizes the strengths and minimize the weaknesses 
associated with single-method designs, while allowing for triangulation of the data (Johnson & Onwuegbuzie, 2004; 
Frechtling & Sharp, 1997). Specifically, a pretest-posttest design (Gall, Borg, & Gall, 1996) was triangulated with 
participant interviews. 

Participants. The participants included an intact group of 16 graduate students (of 19 total) enrolled in an online 
technology integration course during the spring semester of 2006 at a large Midwestern university. Participation in 
the research study was voluntary; the peer feedback process, however, was not voluntary as it was embedded within 
the course. Participants earned extra credit for participating in the study; an alternative extra credit option was also 
presented. 

Participants ranged in age from 24 to 52; 8 were female, 8 were male. Twelve participants were Caucasian, 2 
were African-American, 1 was Hispanic, and 1 identified him/herself as "Other." Seven participants indicated this 
was their first online course, 6 had taken one or two online courses including this course, and 3 had taken more than 
two online courses. Students indicated they spent an average of 6.5 hours per week on the online discussion portion 
of the class. 
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Context and Procedures. The researcher reporting on this exploratory study also served as the faculty member 
teaching the course. While the peer feedback process was embedded in the course the interview data were collected 
by two independent (graduate) researchers and not provided to the researcher until the close of the course and 
assignment of final grades. 

The course, Integration and Management of Computers in Education, was an online, graduate level course. The 
course was co-taught by a faculty member (the researcher) and an experienced graduate assistant who had 
previously taken the course. Students met face-to-face (or via Internet-based video conferencing) for the first class 
session; all subsequent interactions occurred electronically, within a WebCT course management environment. 
While the course had several major projects, this study focused on the weekly discussion questions (DQs) and 
postings. In a typical week, students were given one or two discussion questions and expected to post at least one 
response to each discussion question and two responses to peer's postings for each discussion question. A total of 1 8 
discussion questions were posted over the 15 week semester. Students were also provided with a "free pass" which 
allowed them to skip the discussion question(s) for one week. 

The scoring rubric for the discussion questions was adapted from Ertmer and Stepich (2004), and provided the 
instructors and students with a concrete tool for determining the quality of thinking embedded within online 
postings. Prior to using the rubric for peer feedback students were exposed to modeling of feedback by the 
instructors and examples of possible responses, with an explanation of why each response merited a specific score 
(e.g. handout with examples, discussion in class). Moreover, the course instructors provided two weeks of feedback, 
for a total of four discussion questions. This feedback was provided as a model of the peer feedback the students 
would provide as the course progressed. 

For this study, peer feedback was defined as 1) a numerical score (from 0-2) based on Bloom’s taxonomy and 
2) descriptive comments, supporting the assigned score. Postings demonstrating analysis, synthesis, or evaluation 
received 2 points; postings at the knowledge, comprehension, and application levels received 1 point; non- 
substantive comments received 0 points. The peer feedback process was set up in advance with students receiving 
specific guidelines, including the following: 

• Peer feedback provided was confidential, 

• Peer feedback assignments were randomly assigned for each DQ Students were asked to utilize the Bloom's 
taxonomy rubric to award points in combination with either constructive comments for improvement or a 
rationale for the points awarded. 

• Students were awarded up to 2 points for each peer feedback response completed (8 discussion questions 
requiring peer feedback, two people per question at 2 points each for a total of 32 points); points could be 
deducted if peer feedback was not provided as assigned or in a timely manner. 

• The discussion scores provided via the peer feedback process counted towards students' discussion grades; 
an average of the 2 scores received from peers for each DQ was calculated for an overall DQ score. 

Students were informed that if they felt any score or comment was unwarranted that they could protest the 
feedback via the instructors (mediation process). Only five scores/comments were protested during the 
process. 

In addition, students were provided feedback from the instructor on their use of peer feedback for the first two DQs 
involving the peer feedback process. The process was completed via an online survey system, and channeled 
through the instructors who looked over the comments and scores for any obvious discrepancies or inappropriate 
comments. A sample screenshot of the peer feedback tool is provided (see Figure 1). 
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